[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:32:47 dist-test-slave-65pf krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:32:47 dist-test-slave-65pf krb5kdc[399](info): set up 2 sockets
Sep 25 16:32:47 dist-test-slave-65pf krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260925 16:32:49.083952 392 mini_kdc.cc:121] Time spent starting KDC: real 2.229s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260925 16:32:49.225889 392 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.141s user 0.002s sys 0.000s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260925 16:32:49.428169 392 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.202s user 0.000s sys 0.002s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/krb5kdc/test-user.keytab.
W20260925 16:32:50.131642 392 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.606s user 0.000s sys 0.002s
Sep 25 16:32:50 dist-test-slave-65pf krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:32:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:32:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab.
W20260925 16:32:50.786821 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.62: real 0.631s user 0.003s sys 0.001s
I20260925 16:32:50.787776 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33411
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33411
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:32:50.897219 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:32:50.897481 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:32:50.897521 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:32:50.901445 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:32:50.901535 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:32:50.901556 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:32:50.901572 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:32:50.901587 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:32:50.906910 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33411
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:32:50.907923 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:32:50.908949 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:32:50.917819 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:32:50.917861 412 server_base.cc:1034] running on GCE node
W20260925 16:32:50.917816 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:32:50.917780 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:32:50.918581 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:32:50.919785 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:32:50.920969 412 hybrid_clock.cc:648] HybridClock initialized: now 1790353970920925 us; error 57 us; skew 500 ppm
Sep 25 16:32:50 dist-test-slave-65pf krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353970, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:32:50.923671 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 16:32:50.924575 412 webserver.cc:460] Webserver started at http://127.0.98.62:36477/ using document root <none> and password file <none>
I20260925 16:32:50.925169 412 fs_manager.cc:359] Metadata directory not provided
I20260925 16:32:50.925251 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:32:50.925491 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:32:50.927699 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/minicluster-data/master-0/data/instance:
uuid: "488541c8627f41358aae5a2552dcd4ad"
format_stamp: "Formatted at 2026-09-25 16:32:50 on dist-test-slave-65pf"
server_key: "fe6c3d2d34fdeb9e1d5b6d4543c061b3"
server_key_iv: "031ff0297f12e95f55420153e3f6039f"
server_key_version: "encryptionkey@0"
I20260925 16:32:50.928273 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/minicluster-data/master-0/wal/instance:
uuid: "488541c8627f41358aae5a2552dcd4ad"
format_stamp: "Formatted at 2026-09-25 16:32:50 on dist-test-slave-65pf"
server_key: "fe6c3d2d34fdeb9e1d5b6d4543c061b3"
server_key_iv: "031ff0297f12e95f55420153e3f6039f"
server_key_version: "encryptionkey@0"
I20260925 16:32:50.932032 412 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 16:32:50.934505 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:32:50.935818 412 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:32:50.935945 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/minicluster-data/master-0/wal
uuid: "488541c8627f41358aae5a2552dcd4ad"
format_stamp: "Formatted at 2026-09-25 16:32:50 on dist-test-slave-65pf"
server_key: "fe6c3d2d34fdeb9e1d5b6d4543c061b3"
server_key_iv: "031ff0297f12e95f55420153e3f6039f"
server_key_version: "encryptionkey@0"
I20260925 16:32:50.936053 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:32:50.963043 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:32:50.966153 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:32:50.966367 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:32:50.973317 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33411
I20260925 16:32:50.973338 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33411 every 8 connection(s)
I20260925 16:32:50.974329 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:32:50.977183 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:32:50.982832 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 488541c8627f41358aae5a2552dcd4ad: Bootstrap starting.
I20260925 16:32:50.984186 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 412
I20260925 16:32:50.984323 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/minicluster-data/master-0/wal/instance
I20260925 16:32:50.985571 392 external_mini_cluster.cc:1428] Setting key d44617071ed7c1b43771476f69ea4b99
I20260925 16:32:50.985733 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 488541c8627f41358aae5a2552dcd4ad: Neither blocks nor log segments found. Creating new log.
I20260925 16:32:50.986464 480 log.cc:824] T 00000000000000000000000000000000 P 488541c8627f41358aae5a2552dcd4ad: Log is configured to *not* fsync() on all Append() calls
I20260925 16:32:50.988669 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 488541c8627f41358aae5a2552dcd4ad: No bootstrap required, opened a new log
I20260925 16:32:50.992928 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 488541c8627f41358aae5a2552dcd4ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "488541c8627f41358aae5a2552dcd4ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33411 } }
I20260925 16:32:50.993146 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 488541c8627f41358aae5a2552dcd4ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:32:50.993213 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 488541c8627f41358aae5a2552dcd4ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 488541c8627f41358aae5a2552dcd4ad, State: Initialized, Role: FOLLOWER
I20260925 16:32:50.993655 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 488541c8627f41358aae5a2552dcd4ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "488541c8627f41358aae5a2552dcd4ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33411 } }
I20260925 16:32:50.993778 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 488541c8627f41358aae5a2552dcd4ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:32:50.993857 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 488541c8627f41358aae5a2552dcd4ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:32:50.993971 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 488541c8627f41358aae5a2552dcd4ad [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:32:50.994881 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 488541c8627f41358aae5a2552dcd4ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "488541c8627f41358aae5a2552dcd4ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33411 } }
I20260925 16:32:50.995232 480 leader_election.cc:302] T 00000000000000000000000000000000 P 488541c8627f41358aae5a2552dcd4ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 488541c8627f41358aae5a2552dcd4ad; no voters:
I20260925 16:32:50.995533 480 leader_election.cc:288] T 00000000000000000000000000000000 P 488541c8627f41358aae5a2552dcd4ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:32:50.995626 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 488541c8627f41358aae5a2552dcd4ad [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:32:50.995810 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 488541c8627f41358aae5a2552dcd4ad [term 1 LEADER]: Becoming Leader. State: Replica: 488541c8627f41358aae5a2552dcd4ad, State: Running, Role: LEADER
I20260925 16:32:50.996309 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 488541c8627f41358aae5a2552dcd4ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "488541c8627f41358aae5a2552dcd4ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33411 } }
I20260925 16:32:50.996959 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 488541c8627f41358aae5a2552dcd4ad [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:32:50.997531 485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 488541c8627f41358aae5a2552dcd4ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "488541c8627f41358aae5a2552dcd4ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "488541c8627f41358aae5a2552dcd4ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33411 } } }
I20260925 16:32:50.997740 485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 488541c8627f41358aae5a2552dcd4ad [sys.catalog]: This master's current role is: LEADER
I20260925 16:32:50.997956 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 488541c8627f41358aae5a2552dcd4ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 488541c8627f41358aae5a2552dcd4ad. Latest consensus state: current_term: 1 leader_uuid: "488541c8627f41358aae5a2552dcd4ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "488541c8627f41358aae5a2552dcd4ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33411 } } }
I20260925 16:32:50.998080 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 488541c8627f41358aae5a2552dcd4ad [sys.catalog]: This master's current role is: LEADER
I20260925 16:32:50.998178 491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:32:51.001581 491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 16:32:50 dist-test-slave-65pf krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:32:51.007758 491 catalog_manager.cc:1347] Generated new cluster ID: c90ad6accdea4ddc93b911c485f23622
I20260925 16:32:51.007872 491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:32:51.012678 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:50.988633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59510 (local address 127.0.98.62:33411)
0925 16:32:50.989062 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:50.989080 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:50.990437 (+ 1357us) server_negotiation.cc:407] Connection header received
0925 16:32:50.993222 (+ 2785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:50.993239 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:50.993643 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:50.993988 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:50.997148 (+ 3160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:50.998727 (+ 1579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:51.000557 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:51.001330 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:51.001526 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:51.006884 (+ 5358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:32:51.006892 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:32:51.006906 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:32:51.006953 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:32:51.009060 (+ 2107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:51.009621 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:51.009628 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:51.009633 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:51.009767 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:51.010139 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:51.010142 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:51.010144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:51.010622 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:32:51.010856 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:51.011420 (+ 564us) server_negotiation.cc:299] Negotiation successful
0925 16:32:51.011705 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":143,"threads_started":1}
I20260925 16:32:51.018458 491 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:32:51.019389 491 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:32:51.033231 491 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 488541c8627f41358aae5a2552dcd4ad: Generated new TSK 0
I20260925 16:32:51.034158 491 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 16:32:51.062004 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33411
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:32:51.174772 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:32:51.175045 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:32:51.175089 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:32:51.179201 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:32:51.179261 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:32:51.179322 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 16:32:51.184564 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33411
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:32:51.185576 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:32:51.186571 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:32:51.195920 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:32:51.195937 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:32:51.195924 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:32:51.196214 506 server_base.cc:1034] running on GCE node
I20260925 16:32:51.196764 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:32:51.197451 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:32:51.198657 506 hybrid_clock.cc:648] HybridClock initialized: now 1790353971198629 us; error 62 us; skew 500 ppm
Sep 25 16:32:51 dist-test-slave-65pf krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353971, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:32:51.201435 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 16:32:51.202314 506 webserver.cc:460] Webserver started at http://127.0.98.1:33345/ using document root <none> and password file <none>
I20260925 16:32:51.202936 506 fs_manager.cc:359] Metadata directory not provided
I20260925 16:32:51.203023 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:32:51.203249 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:32:51.205308 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/minicluster-data/ts-0/data/instance:
uuid: "ce2a9c0e9f854f2c91def729d3b7c4c4"
format_stamp: "Formatted at 2026-09-25 16:32:51 on dist-test-slave-65pf"
server_key: "67cb01de2432519a892ce96b6e087579"
server_key_iv: "1512ba34bca643a666eef0f2f49bce40"
server_key_version: "encryptionkey@0"
I20260925 16:32:51.205906 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ce2a9c0e9f854f2c91def729d3b7c4c4"
format_stamp: "Formatted at 2026-09-25 16:32:51 on dist-test-slave-65pf"
server_key: "67cb01de2432519a892ce96b6e087579"
server_key_iv: "1512ba34bca643a666eef0f2f49bce40"
server_key_version: "encryptionkey@0"
I20260925 16:32:51.209767 506 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260925 16:32:51.212373 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:32:51.213656 506 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 16:32:51.213795 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/minicluster-data/ts-0/wal
uuid: "ce2a9c0e9f854f2c91def729d3b7c4c4"
format_stamp: "Formatted at 2026-09-25 16:32:51 on dist-test-slave-65pf"
server_key: "67cb01de2432519a892ce96b6e087579"
server_key_iv: "1512ba34bca643a666eef0f2f49bce40"
server_key_version: "encryptionkey@0"
I20260925 16:32:51.213917 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:32:51.226934 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:32:51.230226 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:32:51.230429 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:32:51.231091 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:32:51.231941 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:32:51.231988 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:32:51.232028 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:32:51.232050 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:32:51.241101 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36967
I20260925 16:32:51.241128 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36967 every 8 connection(s)
I20260925 16:32:51.242027 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
I20260925 16:32:51.248157 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 506
I20260925 16:32:51.248272 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/minicluster-data/ts-0/wal/instance
I20260925 16:32:51.248518 392 external_mini_cluster.cc:1428] Setting key 4de12bf40e187bb0a306c34144225f53
Sep 25 16:32:51 dist-test-slave-65pf krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353971, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:32:51.257066 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:51.244048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38353 (local address 127.0.98.62:33411)
0925 16:32:51.244219 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:51.244224 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:51.245144 (+ 920us) server_negotiation.cc:407] Connection header received
0925 16:32:51.246458 (+ 1314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:51.246461 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:51.246505 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:51.246623 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:51.247785 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:51.248594 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:51.249986 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:51.250218 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:51.250300 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:51.253072 (+ 2772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:32:51.253077 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:32:51.253081 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:32:51.253104 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:32:51.254692 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:51.255334 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:51.255337 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:51.255338 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:51.255409 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:51.255854 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:51.255857 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:51.255858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:51.256063 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:32:51.256152 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:51.256829 (+ 677us) server_negotiation.cc:299] Negotiation successful
0925 16:32:51.256940 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 16:32:51.258344 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:51.244458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33411 (local address 127.0.98.1:38353)
0925 16:32:51.244951 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:51.245011 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0925 16:32:51.246261 (+ 1250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:32:51.246785 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:32:51.246792 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:32:51.247184 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:32:51.247663 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:51.247677 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:51.248752 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:51.248756 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:51.249853 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:51.249863 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:51.250376 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:51.250380 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:51.250633 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:51.251291 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:32:51.251316 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:32:51.252906 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:32:51.254807 (+ 1901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:51.254814 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:51.254833 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:51.255232 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:51.255501 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:51.255507 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:51.255512 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:51.255716 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:51.256180 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:32:51.256188 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:32:51.256556 (+ 368us) client_negotiation.cc:769] Sending connection context
0925 16:32:51.256796 (+ 240us) client_negotiation.cc:240] Negotiation successful
0925 16:32:51.257174 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":119,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 16:32:51.259701 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33411
I20260925 16:32:51.259970 635 heartbeater.cc:464] Registering TS with master...
I20260925 16:32:51.260476 635 heartbeater.cc:511] Master 127.0.98.62:33411 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260925 16:32:51.262086 444 ts_manager.cc:194] Registered new tserver with Master: ce2a9c0e9f854f2c91def729d3b7c4c4 (127.0.98.1:36967)
I20260925 16:32:51.263352 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38353
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 16:32:51.277662 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33411
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:32:51.387315 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:32:51.387583 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:32:51.387627 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:32:51.391621 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:32:51.391677 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:32:51.391734 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 16:32:51.397078 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33411
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:32:51.398216 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:32:51.399260 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:32:51.409171 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:32:51.409198 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:32:51.409171 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:32:51.409487 640 server_base.cc:1034] running on GCE node
I20260925 16:32:51.409881 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:32:51.410490 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:32:51.411657 640 hybrid_clock.cc:648] HybridClock initialized: now 1790353971411647 us; error 33 us; skew 500 ppm
Sep 25 16:32:51 dist-test-slave-65pf krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353971, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:32:51.419317 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 16:32:51.420272 640 webserver.cc:460] Webserver started at http://127.0.98.2:44583/ using document root <none> and password file <none>
I20260925 16:32:51.420838 640 fs_manager.cc:359] Metadata directory not provided
I20260925 16:32:51.420924 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:32:51.421146 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:32:51.423247 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/minicluster-data/ts-1/data/instance:
uuid: "fde33d778db644d1b4ec3d76dec579f9"
format_stamp: "Formatted at 2026-09-25 16:32:51 on dist-test-slave-65pf"
server_key: "be58664db3e3fef059bab59cf3ab9dbb"
server_key_iv: "5d4ef6312e844773ed432bc3115f6118"
server_key_version: "encryptionkey@0"
I20260925 16:32:51.423806 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/minicluster-data/ts-1/wal/instance:
uuid: "fde33d778db644d1b4ec3d76dec579f9"
format_stamp: "Formatted at 2026-09-25 16:32:51 on dist-test-slave-65pf"
server_key: "be58664db3e3fef059bab59cf3ab9dbb"
server_key_iv: "5d4ef6312e844773ed432bc3115f6118"
server_key_version: "encryptionkey@0"
I20260925 16:32:51.427579 640 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 16:32:51.429937 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:32:51.431169 640 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:32:51.431290 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/minicluster-data/ts-1/wal
uuid: "fde33d778db644d1b4ec3d76dec579f9"
format_stamp: "Formatted at 2026-09-25 16:32:51 on dist-test-slave-65pf"
server_key: "be58664db3e3fef059bab59cf3ab9dbb"
server_key_iv: "5d4ef6312e844773ed432bc3115f6118"
server_key_version: "encryptionkey@0"
I20260925 16:32:51.431394 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:32:51.444639 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:32:51.447391 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:32:51.447576 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:32:51.448170 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:32:51.449002 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:32:51.449071 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:32:51.449138 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:32:51.449185 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:32:51.459002 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33323
I20260925 16:32:51.459028 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33323 every 8 connection(s)
I20260925 16:32:51.459877 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
I20260925 16:32:51.463692 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 640
I20260925 16:32:51.463822 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/minicluster-data/ts-1/wal/instance
I20260925 16:32:51.464154 392 external_mini_cluster.cc:1428] Setting key 94724c6799c9d4da73909fb6d981b791
Sep 25 16:32:51 dist-test-slave-65pf krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353971, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260925 16:32:51.474886 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:51.461519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55343 (local address 127.0.98.62:33411)
0925 16:32:51.461673 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:51.461676 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:51.462440 (+ 764us) server_negotiation.cc:407] Connection header received
0925 16:32:51.463927 (+ 1487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:51.463931 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:51.463994 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:51.464093 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:51.465435 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:51.466000 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:51.467635 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:51.468023 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:51.468140 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:51.470966 (+ 2826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:32:51.470969 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:32:51.470971 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:32:51.470994 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:32:51.472448 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:51.473096 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:51.473099 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:51.473100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:51.473181 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:51.473582 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:51.473584 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:51.473585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:51.473818 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:32:51.473896 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:51.474468 (+ 572us) server_negotiation.cc:299] Negotiation successful
0925 16:32:51.474724 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 16:32:51.475560 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:51.461879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33411 (local address 127.0.98.2:55343)
0925 16:32:51.462301 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:51.462341 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 16:32:51.463688 (+ 1347us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:32:51.464320 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:32:51.464331 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:32:51.464738 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:32:51.465293 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:51.465315 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:51.466138 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:51.466145 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:51.467525 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:51.467534 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:51.468168 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:51.468170 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:51.468314 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:51.468913 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:32:51.468928 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:32:51.470799 (+ 1871us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:32:51.472573 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:51.472581 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:51.472602 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:51.472956 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:51.473290 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:51.473293 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:51.473295 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:51.473486 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:51.473918 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:32:51.473926 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:32:51.474213 (+ 287us) client_negotiation.cc:769] Sending connection context
0925 16:32:51.474367 (+ 154us) client_negotiation.cc:240] Negotiation successful
0925 16:32:51.474721 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":107,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260925 16:32:51.476774 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33411
I20260925 16:32:51.476976 769 heartbeater.cc:464] Registering TS with master...
I20260925 16:32:51.477408 769 heartbeater.cc:511] Master 127.0.98.62:33411 requested a full tablet report, sending...
I20260925 16:32:51.478406 444 ts_manager.cc:194] Registered new tserver with Master: fde33d778db644d1b4ec3d76dec579f9 (127.0.98.2:33323)
I20260925 16:32:51.479190 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55343
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 16:32:51.491367 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33411
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:32:51.603566 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:32:51.603822 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:32:51.603866 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:32:51.608460 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:32:51.608522 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:32:51.608597 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 16:32:51.613952 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33411
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:32:51.614918 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:32:51.615897 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:32:51.625468 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:32:51.625484 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:32:51.625494 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:32:51.625672 774 server_base.cc:1034] running on GCE node
I20260925 16:32:51.626222 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:32:51.626950 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:32:51.628118 774 hybrid_clock.cc:648] HybridClock initialized: now 1790353971628095 us; error 35 us; skew 500 ppm
Sep 25 16:32:51 dist-test-slave-65pf krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353971, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:32:51.630822 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 16:32:51.631639 774 webserver.cc:460] Webserver started at http://127.0.98.3:43455/ using document root <none> and password file <none>
I20260925 16:32:51.632184 774 fs_manager.cc:359] Metadata directory not provided
I20260925 16:32:51.632270 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:32:51.632489 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:32:51.634552 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/minicluster-data/ts-2/data/instance:
uuid: "f6f32fb257de430384086632b6478d59"
format_stamp: "Formatted at 2026-09-25 16:32:51 on dist-test-slave-65pf"
server_key: "6ff51a967355242bd335f7e0ba75b33c"
server_key_iv: "fdd4a223be23d3454e64bca5776368d2"
server_key_version: "encryptionkey@0"
I20260925 16:32:51.635223 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f6f32fb257de430384086632b6478d59"
format_stamp: "Formatted at 2026-09-25 16:32:51 on dist-test-slave-65pf"
server_key: "6ff51a967355242bd335f7e0ba75b33c"
server_key_iv: "fdd4a223be23d3454e64bca5776368d2"
server_key_version: "encryptionkey@0"
I20260925 16:32:51.638903 774 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 16:32:51.641222 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:32:51.642387 774 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:32:51.642513 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/minicluster-data/ts-2/wal
uuid: "f6f32fb257de430384086632b6478d59"
format_stamp: "Formatted at 2026-09-25 16:32:51 on dist-test-slave-65pf"
server_key: "6ff51a967355242bd335f7e0ba75b33c"
server_key_iv: "fdd4a223be23d3454e64bca5776368d2"
server_key_version: "encryptionkey@0"
I20260925 16:32:51.642653 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:32:51.652628 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:32:51.655397 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:32:51.655589 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:32:51.656167 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:32:51.656931 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:32:51.657009 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:32:51.657085 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:32:51.657125 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:32:51.666435 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35469
I20260925 16:32:51.666460 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35469 every 8 connection(s)
I20260925 16:32:51.667342 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
I20260925 16:32:51.667639 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 774
I20260925 16:32:51.667745 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353966814071-392-0/minicluster-data/ts-2/wal/instance
I20260925 16:32:51.668052 392 external_mini_cluster.cc:1428] Setting key 45df30bc597f0e01f91fddca905f9916
Sep 25 16:32:51 dist-test-slave-65pf krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353971, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:32:51.680845 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:51.669080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57065 (local address 127.0.98.62:33411)
0925 16:32:51.669207 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:51.669213 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:51.670779 (+ 1566us) server_negotiation.cc:407] Connection header received
0925 16:32:51.672020 (+ 1241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:51.672022 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:51.672066 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:51.672133 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:51.673198 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:51.673703 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:51.674853 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:51.675058 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:51.675135 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:51.677548 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:32:51.677551 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:32:51.677554 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:32:51.677575 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:32:51.678925 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:51.679480 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:51.679483 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:51.679484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:51.679550 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:51.679863 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:51.679866 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:51.679868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:51.680072 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:32:51.680160 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:51.680618 (+ 458us) server_negotiation.cc:299] Negotiation successful
0925 16:32:51.680724 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 16:32:51.681726 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:51.669557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33411 (local address 127.0.98.3:57065)
0925 16:32:51.670647 (+ 1090us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:51.670699 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 16:32:51.671847 (+ 1148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:32:51.672265 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:32:51.672271 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:32:51.672620 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:32:51.673079 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:51.673091 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:51.673799 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:51.673801 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:51.674757 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:51.674765 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:51.675181 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:51.675184 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:51.675315 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:51.675911 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:32:51.675927 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:32:51.677389 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:32:51.679034 (+ 1645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:51.679040 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:51.679052 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:51.679398 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:51.679625 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:51.679627 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:51.679628 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:51.679758 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:51.680163 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:32:51.680168 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:32:51.680400 (+ 232us) client_negotiation.cc:769] Sending connection context
0925 16:32:51.680552 (+ 152us) client_negotiation.cc:240] Negotiation successful
0925 16:32:51.680796 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":846,"thread_start_us":146,"threads_started":1}
I20260925 16:32:51.682884 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33411
I20260925 16:32:51.683107 903 heartbeater.cc:464] Registering TS with master...
I20260925 16:32:51.683609 903 heartbeater.cc:511] Master 127.0.98.62:33411 requested a full tablet report, sending...
I20260925 16:32:51.684652 444 ts_manager.cc:194] Registered new tserver with Master: f6f32fb257de430384086632b6478d59 (127.0.98.3:35469)
I20260925 16:32:51.685281 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57065
I20260925 16:32:51.694147 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 16:32:51 dist-test-slave-65pf krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353971, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 16:32:51 dist-test-slave-65pf krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353971, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 16:32:51.723286 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:51.710461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53118 (local address 127.0.98.1:36967)
0925 16:32:51.710810 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:51.710820 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:51.710844 (+ 24us) server_negotiation.cc:407] Connection header received
0925 16:32:51.710900 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:51.710905 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:51.711090 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:51.711218 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:51.711830 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:51.712586 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:51.713580 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:51.714142 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:51.714253 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:51.718893 (+ 4640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:32:51.718905 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:32:51.718948 (+ 43us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:32:51.718979 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:32:51.721464 (+ 2485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:51.721958 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:51.721963 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:51.721965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:51.722065 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:51.722315 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:51.722317 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:51.722318 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:51.722564 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:32:51.722787 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:51.722977 (+ 190us) server_negotiation.cc:299] Negotiation successful
0925 16:32:51.723132 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":138,"threads_started":1}
W20260925 16:32:51.724264 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53118
Sep 25 16:32:51 dist-test-slave-65pf krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 16:32:51 dist-test-slave-65pf krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 16:32:51.749639 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:51.741655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53134 (local address 127.0.98.1:36967)
0925 16:32:51.741800 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:51.741803 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:51.741820 (+ 17us) server_negotiation.cc:407] Connection header received
0925 16:32:51.741938 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:51.741940 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:51.741980 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:51.742050 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:51.742731 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:51.743236 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:51.744048 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:51.744265 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:51.744340 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:51.746671 (+ 2331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:32:51.746674 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:32:51.746676 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:32:51.746696 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:32:51.748124 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:51.748491 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:51.748494 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:51.748495 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:51.748575 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:51.748824 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:51.748827 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:51.748829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:51.748999 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:32:51.749108 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:51.749293 (+ 185us) server_negotiation.cc:299] Negotiation successful
0925 16:32:51.749392 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 16:32:51.750608 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 506
I20260925 16:32:51.756336 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 640
I20260925 16:32:51.763628 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 774
I20260925 16:32:51.768909 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 412
2026-09-25T16:32:51Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4946 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:32:51 dist-test-slave-65pf krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:32:51 dist-test-slave-65pf krb5kdc[917](info): set up 2 sockets
Sep 25 16:32:51 dist-test-slave-65pf krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260925 16:32:52.798283 392 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/krb5kdc/test-user.keytab.
Sep 25 16:32:52 dist-test-slave-65pf krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:32:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:32:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 16:32:52.919826 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33943
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33943
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:32:53.033726 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:32:53.034075 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:32:53.034144 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:32:53.038267 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:32:53.038368 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:32:53.038419 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:32:53.038457 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:32:53.038509 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:32:53.044056 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33943
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:32:53.045239 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:32:53.046286 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:32:53.054844 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:32:53.054849 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:32:53.054929 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:32:53.055115 930 server_base.cc:1034] running on GCE node
I20260925 16:32:53.055757 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:32:53.056907 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:32:53.058111 930 hybrid_clock.cc:648] HybridClock initialized: now 1790353973058072 us; error 50 us; skew 500 ppm
Sep 25 16:32:53 dist-test-slave-65pf krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:32:53.061076 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 16:32:53.061966 930 webserver.cc:460] Webserver started at http://127.0.98.62:37687/ using document root <none> and password file <none>
I20260925 16:32:53.062539 930 fs_manager.cc:359] Metadata directory not provided
I20260925 16:32:53.062649 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:32:53.062880 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:32:53.064941 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/minicluster-data/master-0/data/instance:
uuid: "72053ad6ee784ba785b1d5114072925e"
format_stamp: "Formatted at 2026-09-25 16:32:53 on dist-test-slave-65pf"
server_key: "e121eed0e6ad22a8703ba86a215e379c"
server_key_iv: "11fefcf398751b41c51c8278ba4034f3"
server_key_version: "encryptionkey@0"
I20260925 16:32:53.065575 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/minicluster-data/master-0/wal/instance:
uuid: "72053ad6ee784ba785b1d5114072925e"
format_stamp: "Formatted at 2026-09-25 16:32:53 on dist-test-slave-65pf"
server_key: "e121eed0e6ad22a8703ba86a215e379c"
server_key_iv: "11fefcf398751b41c51c8278ba4034f3"
server_key_version: "encryptionkey@0"
I20260925 16:32:53.069442 930 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260925 16:32:53.071911 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:32:53.073129 930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 16:32:53.073262 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/minicluster-data/master-0/wal
uuid: "72053ad6ee784ba785b1d5114072925e"
format_stamp: "Formatted at 2026-09-25 16:32:53 on dist-test-slave-65pf"
server_key: "e121eed0e6ad22a8703ba86a215e379c"
server_key_iv: "11fefcf398751b41c51c8278ba4034f3"
server_key_version: "encryptionkey@0"
I20260925 16:32:53.073390 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:32:53.093189 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:32:53.096700 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:32:53.096900 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:32:53.103608 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33943
I20260925 16:32:53.103667 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33943 every 8 connection(s)
I20260925 16:32:53.104575 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:32:53.106178 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 930
I20260925 16:32:53.106304 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/minicluster-data/master-0/wal/instance
I20260925 16:32:53.106539 392 external_mini_cluster.cc:1428] Setting key cb0bc4facc8708825a1182400b741db6
I20260925 16:32:53.107632 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:32:53.114451 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72053ad6ee784ba785b1d5114072925e: Bootstrap starting.
Sep 25 16:32:53 dist-test-slave-65pf krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:32:53.117280 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72053ad6ee784ba785b1d5114072925e: Neither blocks nor log segments found. Creating new log.
I20260925 16:32:53.118000 998 log.cc:824] T 00000000000000000000000000000000 P 72053ad6ee784ba785b1d5114072925e: Log is configured to *not* fsync() on all Append() calls
I20260925 16:32:53.120002 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72053ad6ee784ba785b1d5114072925e: No bootstrap required, opened a new log
I20260925 16:32:53.121748 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:53.107988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56760 (local address 127.0.98.62:33943)
0925 16:32:53.108413 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:53.108435 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:53.108478 (+ 43us) server_negotiation.cc:407] Connection header received
0925 16:32:53.109335 (+ 857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:53.109357 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:53.109790 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:53.110159 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:53.110884 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:53.112350 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:53.113214 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:53.113911 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:53.114058 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:53.116866 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:32:53.116872 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:32:53.116889 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:32:53.116920 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:32:53.118860 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:53.119319 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:53.119324 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:53.119329 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:53.119442 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:53.119723 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:53.119728 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:53.119729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:53.120199 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:32:53.120405 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:53.120685 (+ 280us) server_negotiation.cc:299] Negotiation successful
0925 16:32:53.120994 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":133,"threads_started":1}
I20260925 16:32:53.124032 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72053ad6ee784ba785b1d5114072925e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72053ad6ee784ba785b1d5114072925e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33943 } }
I20260925 16:32:53.124326 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 72053ad6ee784ba785b1d5114072925e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:32:53.124402 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72053ad6ee784ba785b1d5114072925e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72053ad6ee784ba785b1d5114072925e, State: Initialized, Role: FOLLOWER
I20260925 16:32:53.124823 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72053ad6ee784ba785b1d5114072925e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72053ad6ee784ba785b1d5114072925e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33943 } }
I20260925 16:32:53.124971 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 72053ad6ee784ba785b1d5114072925e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:32:53.125051 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72053ad6ee784ba785b1d5114072925e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:32:53.125161 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72053ad6ee784ba785b1d5114072925e [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:32:53.126073 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72053ad6ee784ba785b1d5114072925e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72053ad6ee784ba785b1d5114072925e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33943 } }
I20260925 16:32:53.126410 998 leader_election.cc:302] T 00000000000000000000000000000000 P 72053ad6ee784ba785b1d5114072925e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72053ad6ee784ba785b1d5114072925e; no voters:
I20260925 16:32:53.126760 998 leader_election.cc:288] T 00000000000000000000000000000000 P 72053ad6ee784ba785b1d5114072925e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:32:53.126909 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72053ad6ee784ba785b1d5114072925e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:32:53.127179 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72053ad6ee784ba785b1d5114072925e [term 1 LEADER]: Becoming Leader. State: Replica: 72053ad6ee784ba785b1d5114072925e, State: Running, Role: LEADER
I20260925 16:32:53.127485 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72053ad6ee784ba785b1d5114072925e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72053ad6ee784ba785b1d5114072925e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33943 } }
I20260925 16:32:53.128024 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72053ad6ee784ba785b1d5114072925e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:32:53.128999 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72053ad6ee784ba785b1d5114072925e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72053ad6ee784ba785b1d5114072925e. Latest consensus state: current_term: 1 leader_uuid: "72053ad6ee784ba785b1d5114072925e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72053ad6ee784ba785b1d5114072925e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33943 } } }
I20260925 16:32:53.129127 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72053ad6ee784ba785b1d5114072925e [sys.catalog]: This master's current role is: LEADER
I20260925 16:32:53.129546 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72053ad6ee784ba785b1d5114072925e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "72053ad6ee784ba785b1d5114072925e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72053ad6ee784ba785b1d5114072925e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33943 } } }
I20260925 16:32:53.130213 1012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:32:53.130224 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72053ad6ee784ba785b1d5114072925e [sys.catalog]: This master's current role is: LEADER
I20260925 16:32:53.132830 1012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:32:53.137843 1012 catalog_manager.cc:1347] Generated new cluster ID: f50f485ddb1c421eb92d970b23438d13
I20260925 16:32:53.137926 1012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:32:53.159862 1012 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:32:53.160832 1012 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:32:53.175031 1012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 72053ad6ee784ba785b1d5114072925e: Generated new TSK 0
I20260925 16:32:53.175786 1012 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 16:32:53.211256 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33943
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:32:53.324952 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:32:53.325292 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:32:53.325371 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:32:53.329667 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:32:53.329753 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:32:53.329844 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 16:32:53.335340 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33943
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:32:53.336495 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:32:53.337551 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:32:53.347165 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:32:53.347229 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:32:53.347165 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:32:53.347568 1024 server_base.cc:1034] running on GCE node
I20260925 16:32:53.347973 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:32:53.348645 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:32:53.349906 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790353973349879 us; error 45 us; skew 500 ppm
Sep 25 16:32:53 dist-test-slave-65pf krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:32:53.353010 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 16:32:53.353960 1024 webserver.cc:460] Webserver started at http://127.0.98.1:45553/ using document root <none> and password file <none>
I20260925 16:32:53.354496 1024 fs_manager.cc:359] Metadata directory not provided
I20260925 16:32:53.354557 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:32:53.354796 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:32:53.356833 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/minicluster-data/ts-0/data/instance:
uuid: "3e0a881489fc454382c6f7bdff019689"
format_stamp: "Formatted at 2026-09-25 16:32:53 on dist-test-slave-65pf"
server_key: "3187a4bd979d2618a1faae19e2166c11"
server_key_iv: "f1db9cd17dba1ec1a3578f92de40f914"
server_key_version: "encryptionkey@0"
I20260925 16:32:53.357460 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3e0a881489fc454382c6f7bdff019689"
format_stamp: "Formatted at 2026-09-25 16:32:53 on dist-test-slave-65pf"
server_key: "3187a4bd979d2618a1faae19e2166c11"
server_key_iv: "f1db9cd17dba1ec1a3578f92de40f914"
server_key_version: "encryptionkey@0"
I20260925 16:32:53.361248 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260925 16:32:53.363713 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:32:53.364931 1024 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 16:32:53.365068 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/minicluster-data/ts-0/wal
uuid: "3e0a881489fc454382c6f7bdff019689"
format_stamp: "Formatted at 2026-09-25 16:32:53 on dist-test-slave-65pf"
server_key: "3187a4bd979d2618a1faae19e2166c11"
server_key_iv: "f1db9cd17dba1ec1a3578f92de40f914"
server_key_version: "encryptionkey@0"
I20260925 16:32:53.365170 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:32:53.376351 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:32:53.379678 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:32:53.379887 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:32:53.380474 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:32:53.381258 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:32:53.381302 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:32:53.381378 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:32:53.381417 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:32:53.390193 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36699
I20260925 16:32:53.390236 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36699 every 8 connection(s)
I20260925 16:32:53.391130 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
I20260925 16:32:53.397776 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 1024
I20260925 16:32:53.397887 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/minicluster-data/ts-0/wal/instance
I20260925 16:32:53.398146 392 external_mini_cluster.cc:1428] Setting key 1bad8e97bdb70c328bd08433c83c463b
Sep 25 16:32:53 dist-test-slave-65pf krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:32:53.405555 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:53.392967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52981 (local address 127.0.98.62:33943)
0925 16:32:53.393099 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:53.393108 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:53.393873 (+ 765us) server_negotiation.cc:407] Connection header received
0925 16:32:53.395311 (+ 1438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:53.395314 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:53.395375 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:53.395482 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:53.396631 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:53.397175 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:53.398743 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:53.399039 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:53.399145 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:53.401601 (+ 2456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:32:53.401608 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:32:53.401613 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:32:53.401649 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:32:53.403467 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:53.403955 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:53.403959 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:53.403961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:53.404053 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:53.404358 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:53.404363 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:53.404368 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:53.404632 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:32:53.404746 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:53.405211 (+ 465us) server_negotiation.cc:299] Negotiation successful
0925 16:32:53.405360 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 16:32:53.406322 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:53.393273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33943 (local address 127.0.98.1:52981)
0925 16:32:53.393714 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:53.393758 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 16:32:53.395101 (+ 1343us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:32:53.395651 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:32:53.395658 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:32:53.396011 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:32:53.396490 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:53.396503 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:53.397298 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:53.397301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:53.398620 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:53.398632 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:53.399179 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:53.399182 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:53.399360 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:53.399990 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:32:53.400005 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:32:53.401404 (+ 1399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:32:53.403582 (+ 2178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:53.403588 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:53.403603 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:53.403852 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:53.404150 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:53.404152 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:53.404153 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:53.404271 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:53.404733 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:32:53.404739 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:32:53.404963 (+ 224us) client_negotiation.cc:769] Sending connection context
0925 16:32:53.405112 (+ 149us) client_negotiation.cc:240] Negotiation successful
0925 16:32:53.405391 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":101,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 16:32:53.407627 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33943
I20260925 16:32:53.407924 1153 heartbeater.cc:464] Registering TS with master...
I20260925 16:32:53.408425 1153 heartbeater.cc:511] Master 127.0.98.62:33943 requested a full tablet report, sending...
I20260925 16:32:53.409883 962 ts_manager.cc:194] Registered new tserver with Master: 3e0a881489fc454382c6f7bdff019689 (127.0.98.1:36699)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260925 16:32:53.411484 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52981
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 16:32:53.425688 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33943
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:32:53.535808 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:32:53.536094 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:32:53.536139 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:32:53.540145 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:32:53.540211 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:32:53.540271 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 16:32:53.545462 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33943
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:32:53.546456 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:32:53.547541 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:32:53.557010 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:32:53.557024 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:32:53.557102 1158 server_base.cc:1034] running on GCE node
W20260925 16:32:53.557010 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:32:53.557670 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:32:53.558334 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:32:53.559526 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790353973559497 us; error 41 us; skew 500 ppm
Sep 25 16:32:53 dist-test-slave-65pf krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:32:53.562115 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 16:32:53.563055 1158 webserver.cc:460] Webserver started at http://127.0.98.2:37917/ using document root <none> and password file <none>
I20260925 16:32:53.563611 1158 fs_manager.cc:359] Metadata directory not provided
I20260925 16:32:53.563697 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:32:53.563918 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:32:53.565943 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/minicluster-data/ts-1/data/instance:
uuid: "4c6c3d72073244559c443d3f359b4dab"
format_stamp: "Formatted at 2026-09-25 16:32:53 on dist-test-slave-65pf"
server_key: "f390c21b04fd85796af91a36164a5418"
server_key_iv: "568b31cc77da103ac814881d40c767d0"
server_key_version: "encryptionkey@0"
I20260925 16:32:53.566498 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4c6c3d72073244559c443d3f359b4dab"
format_stamp: "Formatted at 2026-09-25 16:32:53 on dist-test-slave-65pf"
server_key: "f390c21b04fd85796af91a36164a5418"
server_key_iv: "568b31cc77da103ac814881d40c767d0"
server_key_version: "encryptionkey@0"
I20260925 16:32:53.570116 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 16:32:53.572515 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:32:53.573709 1158 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:32:53.573832 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/minicluster-data/ts-1/wal
uuid: "4c6c3d72073244559c443d3f359b4dab"
format_stamp: "Formatted at 2026-09-25 16:32:53 on dist-test-slave-65pf"
server_key: "f390c21b04fd85796af91a36164a5418"
server_key_iv: "568b31cc77da103ac814881d40c767d0"
server_key_version: "encryptionkey@0"
I20260925 16:32:53.573938 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:32:53.612287 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:32:53.615197 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:32:53.615396 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:32:53.615971 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:32:53.616762 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:32:53.616827 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:32:53.616891 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:32:53.616936 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:32:53.627209 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45023
I20260925 16:32:53.627231 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45023 every 8 connection(s)
I20260925 16:32:53.628166 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
I20260925 16:32:53.632261 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 1158
I20260925 16:32:53.632377 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/minicluster-data/ts-1/wal/instance
I20260925 16:32:53.632612 392 external_mini_cluster.cc:1428] Setting key d9bae8312ed7af5340d3301c3c607e32
Sep 25 16:32:53 dist-test-slave-65pf krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260925 16:32:53.642275 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:53.630029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47339 (local address 127.0.98.62:33943)
0925 16:32:53.630140 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:53.630143 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:53.631105 (+ 962us) server_negotiation.cc:407] Connection header received
0925 16:32:53.632428 (+ 1323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:53.632432 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:53.632511 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:53.632609 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:53.633793 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:53.634317 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:53.635489 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:53.635701 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:53.635803 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:53.638489 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:32:53.638492 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:32:53.638494 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:32:53.638515 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:32:53.639957 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:53.640607 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:53.640610 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:53.640611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:53.640693 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:53.641068 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:53.641070 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:53.641071 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:53.641229 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:32:53.641298 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:53.642059 (+ 761us) server_negotiation.cc:299] Negotiation successful
0925 16:32:53.642158 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 16:32:53.643195 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:53.630494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33943 (local address 127.0.98.2:47339)
0925 16:32:53.630957 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:53.631007 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 16:32:53.632225 (+ 1218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:32:53.632820 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:32:53.632827 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:32:53.633168 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:32:53.633648 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:53.633674 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:53.634432 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:53.634435 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:53.635364 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:53.635373 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:53.635837 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:53.635841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:53.636029 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:53.636555 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:32:53.636570 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:32:53.638328 (+ 1758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:32:53.640072 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:53.640078 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:53.640092 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:53.640489 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:53.640786 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:53.640790 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:53.640792 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:53.640967 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:53.641333 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:32:53.641342 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:32:53.641695 (+ 353us) client_negotiation.cc:769] Sending connection context
0925 16:32:53.641913 (+ 218us) client_negotiation.cc:240] Negotiation successful
0925 16:32:53.642271 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":104,"threads_started":1}
I20260925 16:32:53.644400 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33943
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260925 16:32:53.644639 1287 heartbeater.cc:464] Registering TS with master...
I20260925 16:32:53.645109 1287 heartbeater.cc:511] Master 127.0.98.62:33943 requested a full tablet report, sending...
I20260925 16:32:53.646143 962 ts_manager.cc:194] Registered new tserver with Master: 4c6c3d72073244559c443d3f359b4dab (127.0.98.2:45023)
I20260925 16:32:53.646770 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47339
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 16:32:53.661283 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33943
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:32:53.772447 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:32:53.772732 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:32:53.772804 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:32:53.776899 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:32:53.776979 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:32:53.777067 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 16:32:53.782297 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33943
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:32:53.783399 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:32:53.784456 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:32:53.793792 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:32:53.793803 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:32:53.793948 1292 server_base.cc:1034] running on GCE node
W20260925 16:32:53.793803 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:32:53.794387 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:32:53.795054 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:32:53.796203 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790353973796186 us; error 27 us; skew 500 ppm
Sep 25 16:32:53 dist-test-slave-65pf krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:32:53.798808 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 16:32:53.799672 1292 webserver.cc:460] Webserver started at http://127.0.98.3:42991/ using document root <none> and password file <none>
I20260925 16:32:53.800261 1292 fs_manager.cc:359] Metadata directory not provided
I20260925 16:32:53.800343 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:32:53.800565 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:32:53.802548 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/minicluster-data/ts-2/data/instance:
uuid: "7f2047977e854fbf84c55cc888c49813"
format_stamp: "Formatted at 2026-09-25 16:32:53 on dist-test-slave-65pf"
server_key: "346dc48f85d0b85dccbe47eab7505b53"
server_key_iv: "04d6d4e7fda228d41c80ca24fcbe5fd9"
server_key_version: "encryptionkey@0"
I20260925 16:32:53.803144 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7f2047977e854fbf84c55cc888c49813"
format_stamp: "Formatted at 2026-09-25 16:32:53 on dist-test-slave-65pf"
server_key: "346dc48f85d0b85dccbe47eab7505b53"
server_key_iv: "04d6d4e7fda228d41c80ca24fcbe5fd9"
server_key_version: "encryptionkey@0"
I20260925 16:32:53.806852 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 16:32:53.809176 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:32:53.810420 1292 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:32:53.810568 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/minicluster-data/ts-2/wal
uuid: "7f2047977e854fbf84c55cc888c49813"
format_stamp: "Formatted at 2026-09-25 16:32:53 on dist-test-slave-65pf"
server_key: "346dc48f85d0b85dccbe47eab7505b53"
server_key_iv: "04d6d4e7fda228d41c80ca24fcbe5fd9"
server_key_version: "encryptionkey@0"
I20260925 16:32:53.810724 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:32:53.831774 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:32:53.834774 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:32:53.834985 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:32:53.835588 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:32:53.836407 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:32:53.836475 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:32:53.836544 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:32:53.836592 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:32:53.845366 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40325
I20260925 16:32:53.845389 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40325 every 8 connection(s)
I20260925 16:32:53.846285 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
I20260925 16:32:53.847337 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 1292
I20260925 16:32:53.847464 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353966814071-392-0/minicluster-data/ts-2/wal/instance
I20260925 16:32:53.847747 392 external_mini_cluster.cc:1428] Setting key 1e47eea5affa9277e6946dc09d7a7179
Sep 25 16:32:53 dist-test-slave-65pf krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:32:53.859338 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:53.848272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40819 (local address 127.0.98.62:33943)
0925 16:32:53.848395 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:53.848399 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:53.849468 (+ 1069us) server_negotiation.cc:407] Connection header received
0925 16:32:53.850873 (+ 1405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:53.850876 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:53.850926 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:53.850999 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:53.852154 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:53.852711 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:53.853790 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:53.854005 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:53.854080 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:53.856444 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:32:53.856447 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:32:53.856449 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:32:53.856476 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:32:53.857658 (+ 1182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:53.858087 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:53.858089 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:53.858090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:53.858159 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:53.858417 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:53.858419 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:53.858420 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:53.858573 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:32:53.858702 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:53.859124 (+ 422us) server_negotiation.cc:299] Negotiation successful
0925 16:32:53.859220 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 16:32:53.860198 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:53.848781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33943 (local address 127.0.98.3:40819)
0925 16:32:53.849301 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:53.849380 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0925 16:32:53.850693 (+ 1313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:32:53.851150 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:32:53.851156 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:32:53.851512 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:32:53.852009 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:53.852023 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:53.852821 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:53.852824 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:53.853691 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:53.853699 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:53.854120 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:53.854122 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:53.854253 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:53.854903 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:32:53.854919 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:32:53.856304 (+ 1385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:32:53.857751 (+ 1447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:53.857755 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:53.857766 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:53.858002 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:53.858228 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:53.858230 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:53.858232 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:53.858348 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:53.858705 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:32:53.858710 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:32:53.858921 (+ 211us) client_negotiation.cc:769] Sending connection context
0925 16:32:53.859084 (+ 163us) client_negotiation.cc:240] Negotiation successful
0925 16:32:53.859320 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":142,"threads_started":1}
I20260925 16:32:53.861375 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33943
I20260925 16:32:53.861609 1421 heartbeater.cc:464] Registering TS with master...
I20260925 16:32:53.862118 1421 heartbeater.cc:511] Master 127.0.98.62:33943 requested a full tablet report, sending...
I20260925 16:32:53.863121 962 ts_manager.cc:194] Registered new tserver with Master: 7f2047977e854fbf84c55cc888c49813 (127.0.98.3:40325)
I20260925 16:32:53.863730 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40819
I20260925 16:32:53.873697 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 16:32:53.884779 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:53.877444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56772 (local address 127.0.98.62:33943)
0925 16:32:53.877571 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:53.877576 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:53.877735 (+ 159us) server_negotiation.cc:407] Connection header received
0925 16:32:53.877883 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:53.877885 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:53.877929 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:53.878000 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:53.878841 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:53.879482 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:53.880299 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:53.880525 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:53.880613 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:53.881888 (+ 1275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:32:53.881891 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:32:53.881893 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:32:53.881921 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:32:53.883409 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:53.883759 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:53.883762 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:53.883763 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:53.883831 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:53.884034 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:53.884037 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:53.884038 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:53.884233 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:32:53.884333 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:53.884560 (+ 227us) server_negotiation.cc:299] Negotiation successful
0925 16:32:53.884658 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 16:32:53.889904 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56772:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 16:32:53.892962 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:32:53.907826 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:53.901958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41158 (local address 127.0.98.2:45023)
0925 16:32:53.902361 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:53.902372 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:53.902735 (+ 363us) server_negotiation.cc:407] Connection header received
0925 16:32:53.902798 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:53.902801 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:53.902944 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:53.903088 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:32:53.903303 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:53.904182 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:53.906132 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:53.906754 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:53.906868 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:53.907031 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:53.907496 (+ 465us) server_negotiation.cc:299] Negotiation successful
0925 16:32:53.907621 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":115,"threads_started":1}
I20260925 16:32:53.907826 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:53.901113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36699 (local address 127.0.0.1:38340)
0925 16:32:53.901521 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:53.901592 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0925 16:32:53.901712 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:32:53.902468 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:32:53.902473 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:32:53.902503 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:32:53.902681 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:53.902691 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:53.904346 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:53.904349 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:53.905847 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:53.905857 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:53.907074 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:53.907078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:53.907263 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:53.907330 (+ 67us) client_negotiation.cc:769] Sending connection context
0925 16:32:53.907443 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 16:32:53.907624 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":214,"thread_start_us":125,"threads_started":1}
I20260925 16:32:53.907866 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:53.902309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55970 (local address 127.0.98.3:40325)
0925 16:32:53.903114 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:53.903129 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:53.903148 (+ 19us) server_negotiation.cc:407] Connection header received
0925 16:32:53.903214 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:53.903219 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:53.903397 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:53.903539 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:32:53.903763 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:53.904765 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:53.906195 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:53.907005 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:53.907104 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:53.907227 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:53.907627 (+ 400us) server_negotiation.cc:299] Negotiation successful
0925 16:32:53.907721 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":708,"thread_start_us":223,"threads_started":1}
I20260925 16:32:53.908324 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:53.901919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40325 (local address 127.0.0.1:55970)
0925 16:32:53.902803 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:53.902816 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:32:53.902884 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:32:53.903530 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:32:53.903534 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:32:53.903552 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:32:53.903640 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:53.903646 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:53.904984 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:53.904987 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:53.906043 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:53.906054 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:53.907300 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:53.907303 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:53.907467 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:53.907493 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 16:32:53.907581 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 16:32:53.907652 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":820,"mutex_wait_us":83,"thread_start_us":90,"threads_started":1}
I20260925 16:32:53.907838 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:53.901398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45023 (local address 127.0.0.1:41158)
0925 16:32:53.902494 (+ 1096us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:53.902508 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:32:53.902631 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:32:53.903071 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:32:53.903074 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:32:53.903090 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:32:53.903182 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:53.903190 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:53.904294 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:53.904299 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:53.906026 (+ 1727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:53.906038 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:53.907078 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:53.907081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:53.907259 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:53.907347 (+ 88us) client_negotiation.cc:769] Sending connection context
0925 16:32:53.907443 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 16:32:53.907703 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1031,"mutex_wait_us":180,"thread_start_us":101,"threads_started":1}
I20260925 16:32:53.908519 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:53.901304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38340 (local address 127.0.98.1:36699)
0925 16:32:53.901674 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:53.901690 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:53.901716 (+ 26us) server_negotiation.cc:407] Connection header received
0925 16:32:53.902048 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:53.902055 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:53.902277 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:53.902856 (+ 579us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:32:53.903042 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:53.904237 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:53.905983 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:53.906940 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:53.907083 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:53.907227 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:53.908218 (+ 991us) server_negotiation.cc:299] Negotiation successful
0925 16:32:53.908325 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":64,"threads_started":1}
I20260925 16:32:53.911248 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 02e8f6c58d964bb7b07bc27ffbff8972 (DEFAULT_TABLE table=test-table [id=cf50b683c775462d805704f03f12df57]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:32:53.911353 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 02e8f6c58d964bb7b07bc27ffbff8972 (DEFAULT_TABLE table=test-table [id=cf50b683c775462d805704f03f12df57]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:32:53.911353 1355 tablet_service.cc:1467] Processing CreateTablet for tablet 02e8f6c58d964bb7b07bc27ffbff8972 (DEFAULT_TABLE table=test-table [id=cf50b683c775462d805704f03f12df57]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:32:53.912284 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02e8f6c58d964bb7b07bc27ffbff8972. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:32:53.912328 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02e8f6c58d964bb7b07bc27ffbff8972. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:32:53.912328 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02e8f6c58d964bb7b07bc27ffbff8972. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:32:53.918829 1438 tablet_bootstrap.cc:492] T 02e8f6c58d964bb7b07bc27ffbff8972 P 3e0a881489fc454382c6f7bdff019689: Bootstrap starting.
I20260925 16:32:53.919878 1439 tablet_bootstrap.cc:492] T 02e8f6c58d964bb7b07bc27ffbff8972 P 7f2047977e854fbf84c55cc888c49813: Bootstrap starting.
I20260925 16:32:53.920411 1440 tablet_bootstrap.cc:492] T 02e8f6c58d964bb7b07bc27ffbff8972 P 4c6c3d72073244559c443d3f359b4dab: Bootstrap starting.
I20260925 16:32:53.921514 1438 tablet_bootstrap.cc:654] T 02e8f6c58d964bb7b07bc27ffbff8972 P 3e0a881489fc454382c6f7bdff019689: Neither blocks nor log segments found. Creating new log.
I20260925 16:32:53.921960 1439 tablet_bootstrap.cc:654] T 02e8f6c58d964bb7b07bc27ffbff8972 P 7f2047977e854fbf84c55cc888c49813: Neither blocks nor log segments found. Creating new log.
I20260925 16:32:53.922308 1438 log.cc:824] T 02e8f6c58d964bb7b07bc27ffbff8972 P 3e0a881489fc454382c6f7bdff019689: Log is configured to *not* fsync() on all Append() calls
I20260925 16:32:53.922833 1440 tablet_bootstrap.cc:654] T 02e8f6c58d964bb7b07bc27ffbff8972 P 4c6c3d72073244559c443d3f359b4dab: Neither blocks nor log segments found. Creating new log.
I20260925 16:32:53.923332 1439 log.cc:824] T 02e8f6c58d964bb7b07bc27ffbff8972 P 7f2047977e854fbf84c55cc888c49813: Log is configured to *not* fsync() on all Append() calls
I20260925 16:32:53.924075 1440 log.cc:824] T 02e8f6c58d964bb7b07bc27ffbff8972 P 4c6c3d72073244559c443d3f359b4dab: Log is configured to *not* fsync() on all Append() calls
I20260925 16:32:53.925117 1438 tablet_bootstrap.cc:492] T 02e8f6c58d964bb7b07bc27ffbff8972 P 3e0a881489fc454382c6f7bdff019689: No bootstrap required, opened a new log
I20260925 16:32:53.925331 1438 ts_tablet_manager.cc:1397] T 02e8f6c58d964bb7b07bc27ffbff8972 P 3e0a881489fc454382c6f7bdff019689: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260925 16:32:53.925582 1439 tablet_bootstrap.cc:492] T 02e8f6c58d964bb7b07bc27ffbff8972 P 7f2047977e854fbf84c55cc888c49813: No bootstrap required, opened a new log
I20260925 16:32:53.925760 1439 ts_tablet_manager.cc:1397] T 02e8f6c58d964bb7b07bc27ffbff8972 P 7f2047977e854fbf84c55cc888c49813: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 16:32:53.925977 1440 tablet_bootstrap.cc:492] T 02e8f6c58d964bb7b07bc27ffbff8972 P 4c6c3d72073244559c443d3f359b4dab: No bootstrap required, opened a new log
I20260925 16:32:53.926149 1440 ts_tablet_manager.cc:1397] T 02e8f6c58d964bb7b07bc27ffbff8972 P 4c6c3d72073244559c443d3f359b4dab: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260925 16:32:53.929252 1438 raft_consensus.cc:348] T 02e8f6c58d964bb7b07bc27ffbff8972 P 3e0a881489fc454382c6f7bdff019689 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e0a881489fc454382c6f7bdff019689" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36699 } } peers { permanent_uuid: "7f2047977e854fbf84c55cc888c49813" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40325 } } peers { permanent_uuid: "4c6c3d72073244559c443d3f359b4dab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45023 } }
I20260925 16:32:53.929525 1438 raft_consensus.cc:374] T 02e8f6c58d964bb7b07bc27ffbff8972 P 3e0a881489fc454382c6f7bdff019689 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:32:53.929593 1438 raft_consensus.cc:729] T 02e8f6c58d964bb7b07bc27ffbff8972 P 3e0a881489fc454382c6f7bdff019689 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e0a881489fc454382c6f7bdff019689, State: Initialized, Role: FOLLOWER
I20260925 16:32:53.929662 1439 raft_consensus.cc:348] T 02e8f6c58d964bb7b07bc27ffbff8972 P 7f2047977e854fbf84c55cc888c49813 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e0a881489fc454382c6f7bdff019689" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36699 } } peers { permanent_uuid: "7f2047977e854fbf84c55cc888c49813" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40325 } } peers { permanent_uuid: "4c6c3d72073244559c443d3f359b4dab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45023 } }
I20260925 16:32:53.929751 1440 raft_consensus.cc:348] T 02e8f6c58d964bb7b07bc27ffbff8972 P 4c6c3d72073244559c443d3f359b4dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e0a881489fc454382c6f7bdff019689" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36699 } } peers { permanent_uuid: "7f2047977e854fbf84c55cc888c49813" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40325 } } peers { permanent_uuid: "4c6c3d72073244559c443d3f359b4dab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45023 } }
I20260925 16:32:53.929951 1440 raft_consensus.cc:374] T 02e8f6c58d964bb7b07bc27ffbff8972 P 4c6c3d72073244559c443d3f359b4dab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:32:53.929943 1439 raft_consensus.cc:374] T 02e8f6c58d964bb7b07bc27ffbff8972 P 7f2047977e854fbf84c55cc888c49813 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:32:53.930003 1439 raft_consensus.cc:729] T 02e8f6c58d964bb7b07bc27ffbff8972 P 7f2047977e854fbf84c55cc888c49813 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f2047977e854fbf84c55cc888c49813, State: Initialized, Role: FOLLOWER
I20260925 16:32:53.930003 1440 raft_consensus.cc:729] T 02e8f6c58d964bb7b07bc27ffbff8972 P 4c6c3d72073244559c443d3f359b4dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c6c3d72073244559c443d3f359b4dab, State: Initialized, Role: FOLLOWER
I20260925 16:32:53.930116 1438 consensus_queue.cc:260] T 02e8f6c58d964bb7b07bc27ffbff8972 P 3e0a881489fc454382c6f7bdff019689 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e0a881489fc454382c6f7bdff019689" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36699 } } peers { permanent_uuid: "7f2047977e854fbf84c55cc888c49813" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40325 } } peers { permanent_uuid: "4c6c3d72073244559c443d3f359b4dab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45023 } }
I20260925 16:32:53.930480 1440 consensus_queue.cc:260] T 02e8f6c58d964bb7b07bc27ffbff8972 P 4c6c3d72073244559c443d3f359b4dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e0a881489fc454382c6f7bdff019689" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36699 } } peers { permanent_uuid: "7f2047977e854fbf84c55cc888c49813" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40325 } } peers { permanent_uuid: "4c6c3d72073244559c443d3f359b4dab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45023 } }
I20260925 16:32:53.930480 1439 consensus_queue.cc:260] T 02e8f6c58d964bb7b07bc27ffbff8972 P 7f2047977e854fbf84c55cc888c49813 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e0a881489fc454382c6f7bdff019689" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36699 } } peers { permanent_uuid: "7f2047977e854fbf84c55cc888c49813" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40325 } } peers { permanent_uuid: "4c6c3d72073244559c443d3f359b4dab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45023 } }
I20260925 16:32:53.931468 1439 ts_tablet_manager.cc:1428] T 02e8f6c58d964bb7b07bc27ffbff8972 P 7f2047977e854fbf84c55cc888c49813: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 16:32:53.931461 1440 ts_tablet_manager.cc:1428] T 02e8f6c58d964bb7b07bc27ffbff8972 P 4c6c3d72073244559c443d3f359b4dab: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 16:32:53.931535 1438 ts_tablet_manager.cc:1428] T 02e8f6c58d964bb7b07bc27ffbff8972 P 3e0a881489fc454382c6f7bdff019689: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260925 16:32:53.931859 1153 heartbeater.cc:503] Master 127.0.98.62:33943 was elected leader, sending a full tablet report...
I20260925 16:32:53.931838 1421 heartbeater.cc:503] Master 127.0.98.62:33943 was elected leader, sending a full tablet report...
I20260925 16:32:53.932941 1287 heartbeater.cc:503] Master 127.0.98.62:33943 was elected leader, sending a full tablet report...
W20260925 16:32:54.097283 1422 tablet.cc:2369] T 02e8f6c58d964bb7b07bc27ffbff8972 P 7f2047977e854fbf84c55cc888c49813: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 16:32:54.129395 1288 tablet.cc:2369] T 02e8f6c58d964bb7b07bc27ffbff8972 P 4c6c3d72073244559c443d3f359b4dab: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 16:32:54.142563 1154 tablet.cc:2369] T 02e8f6c58d964bb7b07bc27ffbff8972 P 3e0a881489fc454382c6f7bdff019689: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:32:54.157215 1446 raft_consensus.cc:482] T 02e8f6c58d964bb7b07bc27ffbff8972 P 7f2047977e854fbf84c55cc888c49813 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:32:54.157449 1446 raft_consensus.cc:504] T 02e8f6c58d964bb7b07bc27ffbff8972 P 7f2047977e854fbf84c55cc888c49813 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e0a881489fc454382c6f7bdff019689" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36699 } } peers { permanent_uuid: "7f2047977e854fbf84c55cc888c49813" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40325 } } peers { permanent_uuid: "4c6c3d72073244559c443d3f359b4dab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45023 } }
I20260925 16:32:54.158792 1446 leader_election.cc:288] T 02e8f6c58d964bb7b07bc27ffbff8972 P 7f2047977e854fbf84c55cc888c49813 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e0a881489fc454382c6f7bdff019689 (127.0.98.1:36699), 4c6c3d72073244559c443d3f359b4dab (127.0.98.2:45023)
I20260925 16:32:54.162744 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:54.159043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36699 (local address 127.0.98.3:55515)
0925 16:32:54.159171 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:54.159189 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 16:32:54.159290 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:32:54.159558 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:32:54.159561 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:32:54.159578 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:32:54.159694 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:54.159699 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:54.160580 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:54.160583 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:54.161506 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:54.161514 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:54.162261 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:54.162265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:54.162366 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:54.162392 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 16:32:54.162471 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 16:32:54.162556 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 16:32:54.162801 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:54.159125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55515 (local address 127.0.98.1:36699)
0925 16:32:54.159302 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:54.159305 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:54.159321 (+ 16us) server_negotiation.cc:407] Connection header received
0925 16:32:54.159394 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:54.159398 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:54.159444 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:54.159508 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:32:54.159806 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:54.160475 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:54.161623 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:54.162152 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:54.162239 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:54.162288 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:54.162530 (+ 242us) server_negotiation.cc:299] Negotiation successful
0925 16:32:54.162651 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260925 16:32:54.162974 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:54.159146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51585 (local address 127.0.98.2:45023)
0925 16:32:54.159279 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:54.159283 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:54.159824 (+ 541us) server_negotiation.cc:407] Connection header received
0925 16:32:54.159904 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:54.159907 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:54.159949 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:54.160009 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:32:54.160271 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:54.160867 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:54.161953 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:54.162432 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:54.162526 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:54.162606 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:54.162790 (+ 184us) server_negotiation.cc:299] Negotiation successful
0925 16:32:54.162870 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 16:32:54.163203 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:54.159059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45023 (local address 127.0.98.3:51585)
0925 16:32:54.159706 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:54.159717 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 16:32:54.159828 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:32:54.160035 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:32:54.160038 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:32:54.160055 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:32:54.160172 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:54.160177 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:54.160964 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:54.160966 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:54.161837 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:54.161846 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:54.162537 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:54.162541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:54.162658 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:54.162685 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 16:32:54.162760 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 16:32:54.162961 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"mutex_wait_us":313,"thread_start_us":112,"threads_started":1}
I20260925 16:32:54.163566 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02e8f6c58d964bb7b07bc27ffbff8972" candidate_uuid: "7f2047977e854fbf84c55cc888c49813" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e0a881489fc454382c6f7bdff019689" is_pre_election: true
I20260925 16:32:54.163779 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02e8f6c58d964bb7b07bc27ffbff8972" candidate_uuid: "7f2047977e854fbf84c55cc888c49813" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c6c3d72073244559c443d3f359b4dab" is_pre_election: true
I20260925 16:32:54.163911 1107 raft_consensus.cc:2454] T 02e8f6c58d964bb7b07bc27ffbff8972 P 3e0a881489fc454382c6f7bdff019689 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f2047977e854fbf84c55cc888c49813 in term 0.
I20260925 16:32:54.164060 1241 raft_consensus.cc:2454] T 02e8f6c58d964bb7b07bc27ffbff8972 P 4c6c3d72073244559c443d3f359b4dab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f2047977e854fbf84c55cc888c49813 in term 0.
I20260925 16:32:54.164449 1309 leader_election.cc:302] T 02e8f6c58d964bb7b07bc27ffbff8972 P 7f2047977e854fbf84c55cc888c49813 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3e0a881489fc454382c6f7bdff019689, 4c6c3d72073244559c443d3f359b4dab, 7f2047977e854fbf84c55cc888c49813; no voters:
I20260925 16:32:54.164741 1446 raft_consensus.cc:2781] T 02e8f6c58d964bb7b07bc27ffbff8972 P 7f2047977e854fbf84c55cc888c49813 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:32:54.164820 1446 raft_consensus.cc:482] T 02e8f6c58d964bb7b07bc27ffbff8972 P 7f2047977e854fbf84c55cc888c49813 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:32:54.164866 1446 raft_consensus.cc:3037] T 02e8f6c58d964bb7b07bc27ffbff8972 P 7f2047977e854fbf84c55cc888c49813 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:32:54.165920 1446 raft_consensus.cc:504] T 02e8f6c58d964bb7b07bc27ffbff8972 P 7f2047977e854fbf84c55cc888c49813 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e0a881489fc454382c6f7bdff019689" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36699 } } peers { permanent_uuid: "7f2047977e854fbf84c55cc888c49813" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40325 } } peers { permanent_uuid: "4c6c3d72073244559c443d3f359b4dab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45023 } }
I20260925 16:32:54.166280 1446 leader_election.cc:288] T 02e8f6c58d964bb7b07bc27ffbff8972 P 7f2047977e854fbf84c55cc888c49813 [CANDIDATE]: Term 1 election: Requested vote from peers 3e0a881489fc454382c6f7bdff019689 (127.0.98.1:36699), 4c6c3d72073244559c443d3f359b4dab (127.0.98.2:45023)
I20260925 16:32:54.166625 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02e8f6c58d964bb7b07bc27ffbff8972" candidate_uuid: "7f2047977e854fbf84c55cc888c49813" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c6c3d72073244559c443d3f359b4dab"
I20260925 16:32:54.166625 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02e8f6c58d964bb7b07bc27ffbff8972" candidate_uuid: "7f2047977e854fbf84c55cc888c49813" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e0a881489fc454382c6f7bdff019689"
I20260925 16:32:54.166743 1241 raft_consensus.cc:3037] T 02e8f6c58d964bb7b07bc27ffbff8972 P 4c6c3d72073244559c443d3f359b4dab [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:32:54.166743 1107 raft_consensus.cc:3037] T 02e8f6c58d964bb7b07bc27ffbff8972 P 3e0a881489fc454382c6f7bdff019689 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:32:54.167990 1241 raft_consensus.cc:2454] T 02e8f6c58d964bb7b07bc27ffbff8972 P 4c6c3d72073244559c443d3f359b4dab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f2047977e854fbf84c55cc888c49813 in term 1.
I20260925 16:32:54.167994 1107 raft_consensus.cc:2454] T 02e8f6c58d964bb7b07bc27ffbff8972 P 3e0a881489fc454382c6f7bdff019689 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f2047977e854fbf84c55cc888c49813 in term 1.
I20260925 16:32:54.168285 1310 leader_election.cc:302] T 02e8f6c58d964bb7b07bc27ffbff8972 P 7f2047977e854fbf84c55cc888c49813 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c6c3d72073244559c443d3f359b4dab, 7f2047977e854fbf84c55cc888c49813; no voters:
I20260925 16:32:54.168478 1446 raft_consensus.cc:2781] T 02e8f6c58d964bb7b07bc27ffbff8972 P 7f2047977e854fbf84c55cc888c49813 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:32:54.168679 1446 raft_consensus.cc:686] T 02e8f6c58d964bb7b07bc27ffbff8972 P 7f2047977e854fbf84c55cc888c49813 [term 1 LEADER]: Becoming Leader. State: Replica: 7f2047977e854fbf84c55cc888c49813, State: Running, Role: LEADER
I20260925 16:32:54.169034 1446 consensus_queue.cc:237] T 02e8f6c58d964bb7b07bc27ffbff8972 P 7f2047977e854fbf84c55cc888c49813 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e0a881489fc454382c6f7bdff019689" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36699 } } peers { permanent_uuid: "7f2047977e854fbf84c55cc888c49813" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40325 } } peers { permanent_uuid: "4c6c3d72073244559c443d3f359b4dab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45023 } }
I20260925 16:32:54.172200 961 catalog_manager.cc:5495] T 02e8f6c58d964bb7b07bc27ffbff8972 P 7f2047977e854fbf84c55cc888c49813 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f2047977e854fbf84c55cc888c49813 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "7f2047977e854fbf84c55cc888c49813" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e0a881489fc454382c6f7bdff019689" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f2047977e854fbf84c55cc888c49813" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40325 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c6c3d72073244559c443d3f359b4dab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45023 } health_report { overall_health: UNKNOWN } } }
Sep 25 16:32:54 dist-test-slave-65pf krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 16:32:54.201702 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:54.193025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38352 (local address 127.0.98.1:36699)
0925 16:32:54.193193 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:54.193197 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:54.193272 (+ 75us) server_negotiation.cc:407] Connection header received
0925 16:32:54.193379 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:54.193382 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:54.193425 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:54.193494 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:54.194114 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:54.194677 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:54.195463 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:54.195666 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:54.195738 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:54.198245 (+ 2507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:32:54.198257 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:32:54.198272 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:32:54.198297 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:32:54.199984 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:54.200409 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:54.200415 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:54.200416 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:54.200491 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:54.200739 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:54.200743 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:54.200744 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:54.201033 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:32:54.201198 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:54.201346 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 16:32:54.201496 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
Sep 25 16:32:54 dist-test-slave-65pf krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353974, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 16:32:54 dist-test-slave-65pf krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353974, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 16:32:54.228116 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:54.219799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38356 (local address 127.0.98.1:36699)
0925 16:32:54.219989 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:54.219993 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:54.220056 (+ 63us) server_negotiation.cc:407] Connection header received
0925 16:32:54.220178 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:54.220180 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:54.220224 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:54.220300 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:54.221015 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:54.221564 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:54.222346 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:54.222650 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:54.222750 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:54.225119 (+ 2369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:32:54.225122 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:32:54.225124 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:32:54.225149 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:32:54.226466 (+ 1317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:54.226902 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:54.226905 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:54.226906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:54.226986 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:54.227208 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:54.227210 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:54.227211 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:54.227385 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:32:54.227491 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:54.227768 (+ 277us) server_negotiation.cc:299] Negotiation successful
0925 16:32:54.227929 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
Sep 25 16:32:54 dist-test-slave-65pf krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 16:32:54 dist-test-slave-65pf krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 16:32:54.253875 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:54.245266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38368 (local address 127.0.98.1:36699)
0925 16:32:54.245464 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:54.245470 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:54.245566 (+ 96us) server_negotiation.cc:407] Connection header received
0925 16:32:54.245752 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:54.245755 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:54.245797 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:54.245880 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:54.246556 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:54.247130 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:54.247971 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:54.248199 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:54.248305 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:54.250789 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:32:54.250794 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:32:54.250796 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:32:54.250827 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:32:54.252102 (+ 1275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:54.252592 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:54.252596 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:54.252597 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:54.252677 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:54.252962 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:54.252967 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:54.252969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:54.253148 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:32:54.253296 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:54.253550 (+ 254us) server_negotiation.cc:299] Negotiation successful
0925 16:32:54.253683 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260925 16:32:54.255389 1067 tablet_service.cc:2908] Rejecting scan request for tablet 02e8f6c58d964bb7b07bc27ffbff8972: Uninitialized: clean time has not yet been initialized
I20260925 16:32:54.257539 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 1024
I20260925 16:32:54.263688 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 1158
I20260925 16:32:54.269052 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 1292
I20260925 16:32:54.274312 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 930
2026-09-25T16:32:54Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2505 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:32:54 dist-test-slave-65pf krb5kdc[1462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:32:54 dist-test-slave-65pf krb5kdc[1462](info): set up 2 sockets
Sep 25 16:32:54 dist-test-slave-65pf krb5kdc[1462](info): commencing operation
krb5kdc: starting...
W20260925 16:32:55.310485 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/krb5kdc/test-user.keytab.
Sep 25 16:32:55 dist-test-slave-65pf krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:32:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:32:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 16:32:55.429733 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45269
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45269
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:32:55.536448 1475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:32:55.536737 1475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:32:55.536827 1475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:32:55.540788 1475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:32:55.540875 1475 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 16:32:55.540906 1475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:32:55.540954 1475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:32:55.541002 1475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:32:55.541044 1475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:32:55.546229 1475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45269
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:32:55.547286 1475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:32:55.548334 1475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:32:55.556639 1481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:32:55.556587 1480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:32:55.556818 1475 server_base.cc:1034] running on GCE node
W20260925 16:32:55.556995 1483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:32:55.557556 1475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:32:55.558809 1475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:32:55.560019 1475 hybrid_clock.cc:648] HybridClock initialized: now 1790353975559983 us; error 45 us; skew 500 ppm
Sep 25 16:32:55 dist-test-slave-65pf krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:32:55.563036 1475 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 16:32:55.563910 1475 webserver.cc:460] Webserver started at http://127.0.98.62:46071/ using document root <none> and password file <none>
I20260925 16:32:55.564461 1475 fs_manager.cc:359] Metadata directory not provided
I20260925 16:32:55.564543 1475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:32:55.564764 1475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:32:55.566793 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/minicluster-data/master-0/data/instance:
uuid: "eec9855ba8004aecae4b7c47ecce51d3"
format_stamp: "Formatted at 2026-09-25 16:32:55 on dist-test-slave-65pf"
server_key: "0a3d9add0331ebf4fcb5eadcd9132873"
server_key_iv: "0e95d1cf13b9eb558b5d9256453f93f9"
server_key_version: "encryptionkey@0"
I20260925 16:32:55.567337 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/minicluster-data/master-0/wal/instance:
uuid: "eec9855ba8004aecae4b7c47ecce51d3"
format_stamp: "Formatted at 2026-09-25 16:32:55 on dist-test-slave-65pf"
server_key: "0a3d9add0331ebf4fcb5eadcd9132873"
server_key_iv: "0e95d1cf13b9eb558b5d9256453f93f9"
server_key_version: "encryptionkey@0"
I20260925 16:32:55.570930 1475 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 16:32:55.573220 1490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:32:55.574416 1475 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:32:55.574548 1475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/minicluster-data/master-0/wal
uuid: "eec9855ba8004aecae4b7c47ecce51d3"
format_stamp: "Formatted at 2026-09-25 16:32:55 on dist-test-slave-65pf"
server_key: "0a3d9add0331ebf4fcb5eadcd9132873"
server_key_iv: "0e95d1cf13b9eb558b5d9256453f93f9"
server_key_version: "encryptionkey@0"
I20260925 16:32:55.574702 1475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:32:55.601588 1475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:32:55.605080 1475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:32:55.605315 1475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:32:55.613675 1475 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45269
I20260925 16:32:55.613693 1552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45269 every 8 connection(s)
I20260925 16:32:55.614648 1475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:32:55.615804 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 1475
I20260925 16:32:55.615916 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/minicluster-data/master-0/wal/instance
I20260925 16:32:55.616179 392 external_mini_cluster.cc:1428] Setting key 2017b0f7291bc1ded69fc0f6f3390259
I20260925 16:32:55.618042 1553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:32:55.624588 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eec9855ba8004aecae4b7c47ecce51d3: Bootstrap starting.
Sep 25 16:32:55 dist-test-slave-65pf krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:32:55.627101 1553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eec9855ba8004aecae4b7c47ecce51d3: Neither blocks nor log segments found. Creating new log.
I20260925 16:32:55.628217 1553 log.cc:824] T 00000000000000000000000000000000 P eec9855ba8004aecae4b7c47ecce51d3: Log is configured to *not* fsync() on all Append() calls
I20260925 16:32:55.630172 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eec9855ba8004aecae4b7c47ecce51d3: No bootstrap required, opened a new log
I20260925 16:32:55.630198 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:55.617634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52948 (local address 127.0.98.62:45269)
0925 16:32:55.618024 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:55.618041 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:55.618105 (+ 64us) server_negotiation.cc:407] Connection header received
0925 16:32:55.618828 (+ 723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:55.618845 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:55.619167 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:55.619448 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:55.620309 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:55.621271 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:55.622401 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:55.622913 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:55.623028 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:55.625746 (+ 2718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:32:55.625751 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:32:55.625764 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:32:55.625790 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:32:55.627487 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:55.627922 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:55.627932 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:55.627943 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:55.628059 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:55.628342 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:55.628345 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:55.628347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:55.628735 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:32:55.628919 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:55.629222 (+ 303us) server_negotiation.cc:299] Negotiation successful
0925 16:32:55.629492 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":133,"threads_started":1}
I20260925 16:32:55.633517 1553 raft_consensus.cc:348] T 00000000000000000000000000000000 P eec9855ba8004aecae4b7c47ecce51d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eec9855ba8004aecae4b7c47ecce51d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45269 } }
I20260925 16:32:55.633791 1553 raft_consensus.cc:374] T 00000000000000000000000000000000 P eec9855ba8004aecae4b7c47ecce51d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:32:55.633859 1553 raft_consensus.cc:729] T 00000000000000000000000000000000 P eec9855ba8004aecae4b7c47ecce51d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eec9855ba8004aecae4b7c47ecce51d3, State: Initialized, Role: FOLLOWER
I20260925 16:32:55.634321 1553 consensus_queue.cc:260] T 00000000000000000000000000000000 P eec9855ba8004aecae4b7c47ecce51d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eec9855ba8004aecae4b7c47ecce51d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45269 } }
I20260925 16:32:55.634449 1553 raft_consensus.cc:388] T 00000000000000000000000000000000 P eec9855ba8004aecae4b7c47ecce51d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:32:55.634526 1553 raft_consensus.cc:482] T 00000000000000000000000000000000 P eec9855ba8004aecae4b7c47ecce51d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:32:55.634640 1553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eec9855ba8004aecae4b7c47ecce51d3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:32:55.635519 1553 raft_consensus.cc:504] T 00000000000000000000000000000000 P eec9855ba8004aecae4b7c47ecce51d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eec9855ba8004aecae4b7c47ecce51d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45269 } }
I20260925 16:32:55.635877 1553 leader_election.cc:302] T 00000000000000000000000000000000 P eec9855ba8004aecae4b7c47ecce51d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eec9855ba8004aecae4b7c47ecce51d3; no voters:
I20260925 16:32:55.636178 1553 leader_election.cc:288] T 00000000000000000000000000000000 P eec9855ba8004aecae4b7c47ecce51d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:32:55.636270 1558 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eec9855ba8004aecae4b7c47ecce51d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:32:55.636528 1558 raft_consensus.cc:686] T 00000000000000000000000000000000 P eec9855ba8004aecae4b7c47ecce51d3 [term 1 LEADER]: Becoming Leader. State: Replica: eec9855ba8004aecae4b7c47ecce51d3, State: Running, Role: LEADER
I20260925 16:32:55.636806 1558 consensus_queue.cc:237] T 00000000000000000000000000000000 P eec9855ba8004aecae4b7c47ecce51d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eec9855ba8004aecae4b7c47ecce51d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45269 } }
I20260925 16:32:55.637396 1553 sys_catalog.cc:564] T 00000000000000000000000000000000 P eec9855ba8004aecae4b7c47ecce51d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:32:55.638388 1559 sys_catalog.cc:455] T 00000000000000000000000000000000 P eec9855ba8004aecae4b7c47ecce51d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eec9855ba8004aecae4b7c47ecce51d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eec9855ba8004aecae4b7c47ecce51d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45269 } } }
I20260925 16:32:55.638522 1559 sys_catalog.cc:458] T 00000000000000000000000000000000 P eec9855ba8004aecae4b7c47ecce51d3 [sys.catalog]: This master's current role is: LEADER
I20260925 16:32:55.638916 1567 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:32:55.639087 1560 sys_catalog.cc:455] T 00000000000000000000000000000000 P eec9855ba8004aecae4b7c47ecce51d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eec9855ba8004aecae4b7c47ecce51d3. Latest consensus state: current_term: 1 leader_uuid: "eec9855ba8004aecae4b7c47ecce51d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eec9855ba8004aecae4b7c47ecce51d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45269 } } }
I20260925 16:32:55.639298 1560 sys_catalog.cc:458] T 00000000000000000000000000000000 P eec9855ba8004aecae4b7c47ecce51d3 [sys.catalog]: This master's current role is: LEADER
I20260925 16:32:55.641911 1567 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:32:55.646999 1567 catalog_manager.cc:1347] Generated new cluster ID: db392e73793f4eeea9f4e4a086043b49
I20260925 16:32:55.647089 1567 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:32:55.659731 1567 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:32:55.660712 1567 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:32:55.666729 1567 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eec9855ba8004aecae4b7c47ecce51d3: Generated new TSK 0
I20260925 16:32:55.667440 1567 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 16:32:55.705063 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45269
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:32:55.816699 1579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:32:55.817028 1579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:32:55.817106 1579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:32:55.821230 1579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:32:55.821307 1579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:32:55.821380 1579 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 16:32:55.821455 1579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 16:32:55.826745 1579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:32:55.827847 1579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:32:55.828915 1579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:32:55.838446 1584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:32:55.838507 1579 server_base.cc:1034] running on GCE node
W20260925 16:32:55.838456 1587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:32:55.838455 1585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:32:55.839192 1579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:32:55.839884 1579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:32:55.841065 1579 hybrid_clock.cc:648] HybridClock initialized: now 1790353975841050 us; error 33 us; skew 500 ppm
Sep 25 16:32:55 dist-test-slave-65pf krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:32:55.843978 1579 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 16:32:55.844874 1579 webserver.cc:460] Webserver started at http://127.0.98.1:41763/ using document root <none> and password file <none>
I20260925 16:32:55.845435 1579 fs_manager.cc:359] Metadata directory not provided
I20260925 16:32:55.845525 1579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:32:55.845747 1579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:32:55.847813 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/minicluster-data/ts-0/data/instance:
uuid: "5e07d066bb864504b44c19f75dc565b7"
format_stamp: "Formatted at 2026-09-25 16:32:55 on dist-test-slave-65pf"
server_key: "a6c1f545eb8c71a74197d6d6461d2067"
server_key_iv: "73fd27db21a2b027fa58e86ff0247260"
server_key_version: "encryptionkey@0"
I20260925 16:32:55.848398 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5e07d066bb864504b44c19f75dc565b7"
format_stamp: "Formatted at 2026-09-25 16:32:55 on dist-test-slave-65pf"
server_key: "a6c1f545eb8c71a74197d6d6461d2067"
server_key_iv: "73fd27db21a2b027fa58e86ff0247260"
server_key_version: "encryptionkey@0"
I20260925 16:32:55.852223 1579 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 16:32:55.854809 1594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:32:55.856069 1579 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:32:55.856201 1579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/minicluster-data/ts-0/wal
uuid: "5e07d066bb864504b44c19f75dc565b7"
format_stamp: "Formatted at 2026-09-25 16:32:55 on dist-test-slave-65pf"
server_key: "a6c1f545eb8c71a74197d6d6461d2067"
server_key_iv: "73fd27db21a2b027fa58e86ff0247260"
server_key_version: "encryptionkey@0"
I20260925 16:32:55.856345 1579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:32:55.880450 1579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:32:55.883776 1579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:32:55.883984 1579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:32:55.885488 1579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:32:55.885560 1579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:32:55.885630 1579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:32:55.885675 1579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 16:32:55 dist-test-slave-65pf krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:32:55.896281 1579 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44121
I20260925 16:32:55.896551 1709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44121 every 8 connection(s)
I20260925 16:32:55.897454 1579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
I20260925 16:32:55.899686 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:55.886419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39777 (local address 127.0.98.62:45269)
0925 16:32:55.886624 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:55.886629 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:55.887474 (+ 845us) server_negotiation.cc:407] Connection header received
0925 16:32:55.888538 (+ 1064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:55.888541 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:55.888596 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:55.888700 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:55.890277 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:55.891246 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:55.892652 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:55.893090 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:55.893224 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:55.895870 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:32:55.895878 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:32:55.895883 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:32:55.895917 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:32:55.897634 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:55.898310 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:55.898313 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:55.898315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:55.898433 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:55.898768 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:55.898770 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:55.898771 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:55.898950 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:32:55.899026 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:55.899455 (+ 429us) server_negotiation.cc:299] Negotiation successful
0925 16:32:55.899558 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 16:32:55.900429 1603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:55.886875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45269 (local address 127.0.98.1:39777)
0925 16:32:55.887315 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:55.887366 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 16:32:55.888320 (+ 954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:32:55.888880 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:32:55.888893 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:32:55.889428 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:32:55.890102 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:55.890123 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:55.891427 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:55.891430 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:55.892521 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:55.892530 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:55.893253 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:55.893260 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:55.893409 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:55.894006 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:32:55.894022 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:32:55.895697 (+ 1675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:32:55.897860 (+ 2163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:55.897867 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:55.897882 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:55.898219 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:55.898524 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:55.898527 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:55.898528 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:55.898699 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:55.899032 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:32:55.899036 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:32:55.899245 (+ 209us) client_negotiation.cc:769] Sending connection context
0925 16:32:55.899384 (+ 139us) client_negotiation.cc:240] Negotiation successful
0925 16:32:55.899642 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":126,"threads_started":1}
I20260925 16:32:55.901657 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 1579
I20260925 16:32:55.901667 1710 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45269
I20260925 16:32:55.901770 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/minicluster-data/ts-0/wal/instance
I20260925 16:32:55.901801 1710 heartbeater.cc:464] Registering TS with master...
I20260925 16:32:55.902086 392 external_mini_cluster.cc:1428] Setting key 8cebdf6fc1a65b8d6bbdfcfc6c370a4d
I20260925 16:32:55.902338 1710 heartbeater.cc:511] Master 127.0.98.62:45269 requested a full tablet report, sending...
I20260925 16:32:55.903981 1507 ts_manager.cc:194] Registered new tserver with Master: 5e07d066bb864504b44c19f75dc565b7 (127.0.98.1:44121)
I20260925 16:32:55.905618 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39777
I20260925 16:32:55.911369 1716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:55.903965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45269 (local address 127.0.98.1:43305)
0925 16:32:55.904288 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:55.904301 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:32:55.904452 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:32:55.904670 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:32:55.904674 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:32:55.904893 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:32:55.904990 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:55.904995 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:55.905944 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:55.905948 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:55.906777 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:55.906786 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:55.907221 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:55.907225 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:55.907312 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:55.907836 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:32:55.907850 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:32:55.908245 (+ 395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:32:55.909892 (+ 1647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:55.909897 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:55.909899 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:55.910089 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:55.910350 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:55.910355 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:55.910357 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:55.910445 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:55.910836 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:32:55.910839 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:32:55.910926 (+ 87us) client_negotiation.cc:769] Sending connection context
0925 16:32:55.911031 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 16:32:55.911175 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":171,"threads_started":1}
I20260925 16:32:55.911363 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:55.904001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43305 (local address 127.0.98.62:45269)
0925 16:32:55.904121 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:55.904128 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:55.904412 (+ 284us) server_negotiation.cc:407] Connection header received
0925 16:32:55.904527 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:55.904530 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:55.904571 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:55.904657 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:55.905103 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:55.905806 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:55.906887 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:55.907114 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:55.907220 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:55.908360 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:32:55.908364 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:32:55.908367 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:32:55.908389 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:32:55.909785 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:55.910186 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:55.910191 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:55.910193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:55.910267 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:55.910537 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:55.910540 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:55.910542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:55.910746 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:32:55.910848 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:55.911090 (+ 242us) server_negotiation.cc:299] Negotiation successful
0925 16:32:55.911202 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":47}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 16:32:55.930413 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45269
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:32:56.044296 1719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:32:56.044597 1719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:32:56.044687 1719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:32:56.048767 1719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:32:56.048859 1719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:32:56.048926 1719 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 16:32:56.049005 1719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 16:32:56.054296 1719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:32:56.055436 1719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:32:56.056499 1719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:32:56.066046 1724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:32:56.066046 1727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:32:56.066068 1725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:32:56.066393 1719 server_base.cc:1034] running on GCE node
I20260925 16:32:56.066860 1719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:32:56.067709 1719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:32:56.068877 1719 hybrid_clock.cc:648] HybridClock initialized: now 1790353976068860 us; error 33 us; skew 500 ppm
Sep 25 16:32:56 dist-test-slave-65pf krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353976, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:32:56.071699 1719 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 16:32:56.072600 1719 webserver.cc:460] Webserver started at http://127.0.98.2:42937/ using document root <none> and password file <none>
I20260925 16:32:56.073258 1719 fs_manager.cc:359] Metadata directory not provided
I20260925 16:32:56.073354 1719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:32:56.073583 1719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:32:56.075693 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/minicluster-data/ts-1/data/instance:
uuid: "f1e3c666b63f43c796658fd44d19244e"
format_stamp: "Formatted at 2026-09-25 16:32:56 on dist-test-slave-65pf"
server_key: "2314b1ba7692b69c446e11d3682f343a"
server_key_iv: "8997ee87cbd751fe9e45ac896048db01"
server_key_version: "encryptionkey@0"
I20260925 16:32:56.076246 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f1e3c666b63f43c796658fd44d19244e"
format_stamp: "Formatted at 2026-09-25 16:32:56 on dist-test-slave-65pf"
server_key: "2314b1ba7692b69c446e11d3682f343a"
server_key_iv: "8997ee87cbd751fe9e45ac896048db01"
server_key_version: "encryptionkey@0"
I20260925 16:32:56.080089 1719 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 16:32:56.082463 1734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:32:56.083722 1719 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:32:56.083837 1719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/minicluster-data/ts-1/wal
uuid: "f1e3c666b63f43c796658fd44d19244e"
format_stamp: "Formatted at 2026-09-25 16:32:56 on dist-test-slave-65pf"
server_key: "2314b1ba7692b69c446e11d3682f343a"
server_key_iv: "8997ee87cbd751fe9e45ac896048db01"
server_key_version: "encryptionkey@0"
I20260925 16:32:56.083935 1719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:32:56.120059 1719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:32:56.123199 1719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:32:56.123420 1719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:32:56.124924 1719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:32:56.124972 1719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:32:56.125038 1719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:32:56.125070 1719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 16:32:56 dist-test-slave-65pf krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353976, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:32:56.135872 1719 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46439
I20260925 16:32:56.135946 1849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46439 every 8 connection(s)
I20260925 16:32:56.137082 1719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
I20260925 16:32:56.139590 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:56.125835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44637 (local address 127.0.98.62:45269)
0925 16:32:56.125959 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:56.125963 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:56.127098 (+ 1135us) server_negotiation.cc:407] Connection header received
0925 16:32:56.128205 (+ 1107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:56.128210 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:56.128276 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:56.128353 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:56.130193 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:56.131142 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:56.132509 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:56.132790 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:56.132900 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:56.135570 (+ 2670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:32:56.135576 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:32:56.135582 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:32:56.135619 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:32:56.137314 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:56.137931 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:56.137934 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:56.137936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:56.138034 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:56.138404 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:56.138408 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:56.138410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:56.138729 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:32:56.138826 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:56.139297 (+ 471us) server_negotiation.cc:299] Negotiation successful
0925 16:32:56.139423 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 16:32:56.140406 1743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:56.126433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45269 (local address 127.0.98.2:44637)
0925 16:32:56.126954 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:56.126996 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 16:32:56.127920 (+ 924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:32:56.128796 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:32:56.128807 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:32:56.129314 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:32:56.130034 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:56.130057 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:56.131283 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:56.131288 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:56.132372 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:56.132381 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:56.132952 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:56.132957 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:56.133131 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:56.133766 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:32:56.133782 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:32:56.135363 (+ 1581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:32:56.137475 (+ 2112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:56.137482 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:56.137502 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:56.137831 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:56.138128 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:56.138131 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:56.138133 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:56.138294 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:56.138821 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:32:56.138827 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:32:56.139067 (+ 240us) client_negotiation.cc:769] Sending connection context
0925 16:32:56.139244 (+ 177us) client_negotiation.cc:240] Negotiation successful
0925 16:32:56.139601 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":169,"threads_started":1}
I20260925 16:32:56.141654 1850 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45269
I20260925 16:32:56.141837 1850 heartbeater.cc:464] Registering TS with master...
I20260925 16:32:56.142465 1850 heartbeater.cc:511] Master 127.0.98.62:45269 requested a full tablet report, sending...
I20260925 16:32:56.143567 1505 ts_manager.cc:194] Registered new tserver with Master: f1e3c666b63f43c796658fd44d19244e (127.0.98.2:46439)
I20260925 16:32:56.144416 1505 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44637
I20260925 16:32:56.147105 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 1719
I20260925 16:32:56.147218 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/minicluster-data/ts-1/wal/instance
I20260925 16:32:56.147490 392 external_mini_cluster.cc:1428] Setting key 093e9b905cb89cb66e443bf942051e10
I20260925 16:32:56.151118 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:56.143748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55375 (local address 127.0.98.62:45269)
0925 16:32:56.143869 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:56.143873 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:56.144059 (+ 186us) server_negotiation.cc:407] Connection header received
0925 16:32:56.144182 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:56.144184 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:56.144218 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:56.144279 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:56.144768 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:56.145476 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:56.146283 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:56.146533 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:56.146689 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:56.147905 (+ 1216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:32:56.147913 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:32:56.147915 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:32:56.147952 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:32:56.149506 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:56.149920 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:56.149925 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:56.149930 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:56.150012 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:56.150272 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:56.150276 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:56.150278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:56.150485 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:32:56.150588 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:56.150860 (+ 272us) server_negotiation.cc:299] Negotiation successful
0925 16:32:56.150965 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 16:32:56.151141 1856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:56.143691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45269 (local address 127.0.98.2:55375)
0925 16:32:56.143988 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:56.144003 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:32:56.144102 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:32:56.144318 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:32:56.144321 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:32:56.144530 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:32:56.144650 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:56.144661 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:56.145597 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:56.145599 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:56.146169 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:56.146176 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:56.146703 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:56.146706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:56.146770 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:56.147355 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:32:56.147366 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:32:56.147745 (+ 379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:32:56.149631 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:56.149634 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:56.149636 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:56.149826 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:56.150104 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:56.150107 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:56.150109 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:56.150189 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:56.150599 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:32:56.150603 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:32:56.150700 (+ 97us) client_negotiation.cc:769] Sending connection context
0925 16:32:56.150799 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 16:32:56.150954 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":111,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 16:32:56.174023 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45269
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:32:56.284511 1859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:32:56.284776 1859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:32:56.284857 1859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:32:56.289006 1859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:32:56.289069 1859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:32:56.289139 1859 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 16:32:56.289209 1859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 16:32:56.294562 1859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:32:56.295742 1859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:32:56.296851 1859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:32:56.306617 1865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:32:56.306617 1867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:32:56.306865 1859 server_base.cc:1034] running on GCE node
W20260925 16:32:56.306929 1864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:32:56.307324 1859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:32:56.308004 1859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:32:56.309163 1859 hybrid_clock.cc:648] HybridClock initialized: now 1790353976309146 us; error 28 us; skew 500 ppm
Sep 25 16:32:56 dist-test-slave-65pf krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353976, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:32:56.312021 1859 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 16:32:56.312865 1859 webserver.cc:460] Webserver started at http://127.0.98.3:40947/ using document root <none> and password file <none>
I20260925 16:32:56.313402 1859 fs_manager.cc:359] Metadata directory not provided
I20260925 16:32:56.313462 1859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:32:56.313629 1859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:32:56.315713 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/minicluster-data/ts-2/data/instance:
uuid: "02ab510d1c094169b31adb687a154454"
format_stamp: "Formatted at 2026-09-25 16:32:56 on dist-test-slave-65pf"
server_key: "cbe87324df2cb599a380b1fca4f6cbfd"
server_key_iv: "14b73b959e45b5db4b6c5e3b1114b9a6"
server_key_version: "encryptionkey@0"
I20260925 16:32:56.316213 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/minicluster-data/ts-2/wal/instance:
uuid: "02ab510d1c094169b31adb687a154454"
format_stamp: "Formatted at 2026-09-25 16:32:56 on dist-test-slave-65pf"
server_key: "cbe87324df2cb599a380b1fca4f6cbfd"
server_key_iv: "14b73b959e45b5db4b6c5e3b1114b9a6"
server_key_version: "encryptionkey@0"
I20260925 16:32:56.319948 1859 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 16:32:56.322471 1874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:32:56.323767 1859 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:32:56.323901 1859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/minicluster-data/ts-2/wal
uuid: "02ab510d1c094169b31adb687a154454"
format_stamp: "Formatted at 2026-09-25 16:32:56 on dist-test-slave-65pf"
server_key: "cbe87324df2cb599a380b1fca4f6cbfd"
server_key_iv: "14b73b959e45b5db4b6c5e3b1114b9a6"
server_key_version: "encryptionkey@0"
I20260925 16:32:56.324018 1859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:32:56.337261 1859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:32:56.340884 1859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:32:56.341107 1859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:32:56.342792 1859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:32:56.342888 1859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:32:56.342979 1859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:32:56.343014 1859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 16:32:56 dist-test-slave-65pf krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353976, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:32:56.353940 1859 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42469
I20260925 16:32:56.355264 1859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
I20260925 16:32:56.355310 1989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42469 every 8 connection(s)
I20260925 16:32:56.358697 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:56.343801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46183 (local address 127.0.98.62:45269)
0925 16:32:56.343930 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:56.343934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:56.345014 (+ 1080us) server_negotiation.cc:407] Connection header received
0925 16:32:56.346264 (+ 1250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:56.346271 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:56.346345 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:56.346437 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:56.348334 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:56.349236 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:56.350573 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:56.350951 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:56.351067 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:56.354108 (+ 3041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:32:56.354115 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:32:56.354119 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:32:56.354152 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:32:56.356099 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:56.356775 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:56.356783 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:56.356786 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:56.356914 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:56.357358 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:56.357363 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:56.357365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:56.357656 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:32:56.357769 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:56.358385 (+ 616us) server_negotiation.cc:299] Negotiation successful
0925 16:32:56.358509 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 16:32:56.359844 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:56.344347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45269 (local address 127.0.98.3:46183)
0925 16:32:56.344848 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:56.344891 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 16:32:56.345988 (+ 1097us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:32:56.346820 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:32:56.346828 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:32:56.347342 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:32:56.348143 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:56.348167 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:56.349402 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:56.349405 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:56.350446 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:56.350455 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:56.351106 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:56.351110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:56.351287 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:56.352003 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:32:56.352026 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:32:56.353893 (+ 1867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:32:56.356230 (+ 2337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:56.356240 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:56.356257 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:56.356641 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:56.357053 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:56.357056 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:56.357058 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:56.357251 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:56.357764 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:32:56.357798 (+ 34us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:32:56.358125 (+ 327us) client_negotiation.cc:769] Sending connection context
0925 16:32:56.358334 (+ 209us) client_negotiation.cc:240] Negotiation successful
0925 16:32:56.358695 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":136,"threads_started":1}
I20260925 16:32:56.359975 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 1859
I20260925 16:32:56.360085 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/minicluster-data/ts-2/wal/instance
I20260925 16:32:56.360528 392 external_mini_cluster.cc:1428] Setting key e1c2590ef5069fb389aa9bd68edce1d7
I20260925 16:32:56.361512 1990 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45269
I20260925 16:32:56.361672 1990 heartbeater.cc:464] Registering TS with master...
I20260925 16:32:56.362207 1990 heartbeater.cc:511] Master 127.0.98.62:45269 requested a full tablet report, sending...
I20260925 16:32:56.363812 1507 ts_manager.cc:194] Registered new tserver with Master: 02ab510d1c094169b31adb687a154454 (127.0.98.3:42469)
I20260925 16:32:56.364396 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46183
I20260925 16:32:56.365949 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 16:32:56.371337 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:56.363817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47603 (local address 127.0.98.62:45269)
0925 16:32:56.363936 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:56.363941 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:56.364257 (+ 316us) server_negotiation.cc:407] Connection header received
0925 16:32:56.364385 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:56.364388 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:56.364431 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:56.364517 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:56.365065 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:56.365607 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:56.366691 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:56.366928 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:56.367036 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:56.368136 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:32:56.368142 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:32:56.368145 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:32:56.368178 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:32:56.369732 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:56.370135 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:56.370140 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:56.370142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:56.370224 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:56.370491 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:56.370496 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:56.370498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:56.370721 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:32:56.370824 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:56.371077 (+ 253us) server_negotiation.cc:299] Negotiation successful
0925 16:32:56.371187 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 16:32:56.371348 1996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:56.363904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45269 (local address 127.0.98.3:47603)
0925 16:32:56.364171 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:56.364184 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:32:56.364296 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:32:56.364530 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:32:56.364533 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:32:56.364763 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:32:56.364899 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:56.364907 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:56.365742 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:56.365745 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:56.366527 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:56.366536 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:56.367041 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:56.367044 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:56.367131 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:56.367656 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:32:56.367670 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:32:56.367998 (+ 328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:32:56.369837 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:56.369842 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:56.369844 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:56.370042 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:56.370316 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:56.370320 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:56.370322 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:56.370407 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:56.370816 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:32:56.370819 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:32:56.370912 (+ 93us) client_negotiation.cc:769] Sending connection context
0925 16:32:56.371014 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 16:32:56.371159 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":119,"threads_started":1}
Sep 25 16:32:56 dist-test-slave-65pf krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353976, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 16:32:56 dist-test-slave-65pf krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353976, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:32:56.392138 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:56.383889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52950 (local address 127.0.98.62:45269)
0925 16:32:56.384013 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:56.384016 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:56.384028 (+ 12us) server_negotiation.cc:407] Connection header received
0925 16:32:56.384065 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:56.384067 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:56.384105 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:56.384172 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:56.384823 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:56.385330 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:56.386279 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:56.386698 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:56.386837 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:56.389244 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:32:56.389247 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:32:56.389249 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:32:56.389272 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:32:56.390732 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:56.391094 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:56.391096 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:56.391098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:56.391167 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:56.391370 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:56.391372 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:56.391373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:56.391536 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:32:56.391630 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:56.391818 (+ 188us) server_negotiation.cc:299] Negotiation successful
0925 16:32:56.391921 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 16:32:56.394949 1505 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52950:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 16:32:56.397863 1505 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:32:56.411881 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:56.406712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47804 (local address 127.0.98.1:44121)
0925 16:32:56.407484 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:56.407490 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:56.407510 (+ 20us) server_negotiation.cc:407] Connection header received
0925 16:32:56.407572 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:56.407576 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:56.407727 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:56.408009 (+ 282us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:32:56.408158 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:56.408854 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:56.410373 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:56.411179 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:56.411298 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:56.411460 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:56.411644 (+ 184us) server_negotiation.cc:299] Negotiation successful
0925 16:32:56.411742 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":81,"threads_started":1}
I20260925 16:32:56.411882 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:56.405893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46439 (local address 127.0.0.1:56058)
0925 16:32:56.406972 (+ 1079us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:56.406985 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:32:56.407081 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:32:56.407603 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:32:56.407610 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:32:56.407637 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:32:56.407838 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:56.407845 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:56.409353 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:56.409357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:56.410447 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:56.410454 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:56.411443 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:56.411447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:56.411595 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:56.411622 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 16:32:56.411674 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 16:32:56.411742 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":978,"mutex_wait_us":15,"thread_start_us":80,"threads_started":1}
I20260925 16:32:56.411888 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:56.405816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44121 (local address 127.0.0.1:47804)
0925 16:32:56.406257 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:56.406406 (+ 149us) client_negotiation.cc:174] Beginning negotiation
0925 16:32:56.406539 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:32:56.407861 (+ 1322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:32:56.407864 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:32:56.407884 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:32:56.408061 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:56.408068 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:56.408966 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:56.408971 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:56.410255 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:56.410265 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:56.411305 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:56.411312 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:56.411440 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:56.411484 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 16:32:56.411600 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 16:32:56.411727 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":41,"thread_start_us":68,"threads_started":1}
I20260925 16:32:56.412597 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:56.406611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56058 (local address 127.0.98.2:46439)
0925 16:32:56.406908 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:56.406920 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:56.407055 (+ 135us) server_negotiation.cc:407] Connection header received
0925 16:32:56.407190 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:56.407199 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:56.407427 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:56.407879 (+ 452us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:32:56.408033 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:56.409239 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:56.410557 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:56.411315 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:56.412215 (+ 900us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:56.412329 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:56.412367 (+ 38us) server_negotiation.cc:299] Negotiation successful
0925 16:32:56.412456 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":80,"threads_started":1}
I20260925 16:32:56.413033 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:56.406522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42469 (local address 127.0.0.1:55070)
0925 16:32:56.407071 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:56.407086 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:32:56.407166 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:32:56.407733 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:32:56.407735 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:32:56.407747 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:32:56.408014 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:56.408022 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:56.410387 (+ 2365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:56.410389 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:56.411228 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:56.411256 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:56.412653 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:56.412657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:56.412773 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:56.412797 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 16:32:56.412857 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 16:32:56.412919 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"mutex_wait_us":24,"thread_start_us":425,"threads_started":1}
I20260925 16:32:56.413964 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:56.406726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55070 (local address 127.0.98.3:42469)
0925 16:32:56.407328 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:56.407341 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:56.407363 (+ 22us) server_negotiation.cc:407] Connection header received
0925 16:32:56.407432 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:56.407437 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:56.407604 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:56.407762 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:32:56.408142 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:56.409239 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:56.411383 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:56.412543 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:56.413550 (+ 1007us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:56.413696 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:56.413745 (+ 49us) server_negotiation.cc:299] Negotiation successful
0925 16:32:56.413825 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":116,"threads_started":1}
I20260925 16:32:56.415205 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 0d5ad44c65da4013a7ea47dc61d7381f (DEFAULT_TABLE table=test-table [id=d569bf9030864380b5e4179b249e4b43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:32:56.415239 1784 tablet_service.cc:1467] Processing CreateTablet for tablet 0d5ad44c65da4013a7ea47dc61d7381f (DEFAULT_TABLE table=test-table [id=d569bf9030864380b5e4179b249e4b43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:32:56.416163 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d5ad44c65da4013a7ea47dc61d7381f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:32:56.416218 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d5ad44c65da4013a7ea47dc61d7381f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:32:56.416468 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 0d5ad44c65da4013a7ea47dc61d7381f (DEFAULT_TABLE table=test-table [id=d569bf9030864380b5e4179b249e4b43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:32:56.417336 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d5ad44c65da4013a7ea47dc61d7381f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:32:56.421916 2013 tablet_bootstrap.cc:492] T 0d5ad44c65da4013a7ea47dc61d7381f P 5e07d066bb864504b44c19f75dc565b7: Bootstrap starting.
I20260925 16:32:56.424530 2013 tablet_bootstrap.cc:654] T 0d5ad44c65da4013a7ea47dc61d7381f P 5e07d066bb864504b44c19f75dc565b7: Neither blocks nor log segments found. Creating new log.
I20260925 16:32:56.425269 2013 log.cc:824] T 0d5ad44c65da4013a7ea47dc61d7381f P 5e07d066bb864504b44c19f75dc565b7: Log is configured to *not* fsync() on all Append() calls
I20260925 16:32:56.425823 2014 tablet_bootstrap.cc:492] T 0d5ad44c65da4013a7ea47dc61d7381f P f1e3c666b63f43c796658fd44d19244e: Bootstrap starting.
I20260925 16:32:56.426872 2015 tablet_bootstrap.cc:492] T 0d5ad44c65da4013a7ea47dc61d7381f P 02ab510d1c094169b31adb687a154454: Bootstrap starting.
I20260925 16:32:56.427223 2013 tablet_bootstrap.cc:492] T 0d5ad44c65da4013a7ea47dc61d7381f P 5e07d066bb864504b44c19f75dc565b7: No bootstrap required, opened a new log
I20260925 16:32:56.427413 2013 ts_tablet_manager.cc:1397] T 0d5ad44c65da4013a7ea47dc61d7381f P 5e07d066bb864504b44c19f75dc565b7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 16:32:56.428012 2014 tablet_bootstrap.cc:654] T 0d5ad44c65da4013a7ea47dc61d7381f P f1e3c666b63f43c796658fd44d19244e: Neither blocks nor log segments found. Creating new log.
I20260925 16:32:56.428754 2015 tablet_bootstrap.cc:654] T 0d5ad44c65da4013a7ea47dc61d7381f P 02ab510d1c094169b31adb687a154454: Neither blocks nor log segments found. Creating new log.
I20260925 16:32:56.428977 2014 log.cc:824] T 0d5ad44c65da4013a7ea47dc61d7381f P f1e3c666b63f43c796658fd44d19244e: Log is configured to *not* fsync() on all Append() calls
I20260925 16:32:56.429510 2015 log.cc:824] T 0d5ad44c65da4013a7ea47dc61d7381f P 02ab510d1c094169b31adb687a154454: Log is configured to *not* fsync() on all Append() calls
I20260925 16:32:56.430874 2014 tablet_bootstrap.cc:492] T 0d5ad44c65da4013a7ea47dc61d7381f P f1e3c666b63f43c796658fd44d19244e: No bootstrap required, opened a new log
I20260925 16:32:56.431070 2014 ts_tablet_manager.cc:1397] T 0d5ad44c65da4013a7ea47dc61d7381f P f1e3c666b63f43c796658fd44d19244e: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260925 16:32:56.431185 2015 tablet_bootstrap.cc:492] T 0d5ad44c65da4013a7ea47dc61d7381f P 02ab510d1c094169b31adb687a154454: No bootstrap required, opened a new log
I20260925 16:32:56.431353 2015 ts_tablet_manager.cc:1397] T 0d5ad44c65da4013a7ea47dc61d7381f P 02ab510d1c094169b31adb687a154454: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260925 16:32:56.431738 2013 raft_consensus.cc:348] T 0d5ad44c65da4013a7ea47dc61d7381f P 5e07d066bb864504b44c19f75dc565b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e07d066bb864504b44c19f75dc565b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44121 } } peers { permanent_uuid: "f1e3c666b63f43c796658fd44d19244e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46439 } } peers { permanent_uuid: "02ab510d1c094169b31adb687a154454" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42469 } }
I20260925 16:32:56.432008 2013 raft_consensus.cc:374] T 0d5ad44c65da4013a7ea47dc61d7381f P 5e07d066bb864504b44c19f75dc565b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:32:56.432065 2013 raft_consensus.cc:729] T 0d5ad44c65da4013a7ea47dc61d7381f P 5e07d066bb864504b44c19f75dc565b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e07d066bb864504b44c19f75dc565b7, State: Initialized, Role: FOLLOWER
I20260925 16:32:56.432508 2013 consensus_queue.cc:260] T 0d5ad44c65da4013a7ea47dc61d7381f P 5e07d066bb864504b44c19f75dc565b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e07d066bb864504b44c19f75dc565b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44121 } } peers { permanent_uuid: "f1e3c666b63f43c796658fd44d19244e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46439 } } peers { permanent_uuid: "02ab510d1c094169b31adb687a154454" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42469 } }
I20260925 16:32:56.433142 1710 heartbeater.cc:503] Master 127.0.98.62:45269 was elected leader, sending a full tablet report...
I20260925 16:32:56.433468 2013 ts_tablet_manager.cc:1428] T 0d5ad44c65da4013a7ea47dc61d7381f P 5e07d066bb864504b44c19f75dc565b7: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 16:32:56.434664 2014 raft_consensus.cc:348] T 0d5ad44c65da4013a7ea47dc61d7381f P f1e3c666b63f43c796658fd44d19244e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e07d066bb864504b44c19f75dc565b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44121 } } peers { permanent_uuid: "f1e3c666b63f43c796658fd44d19244e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46439 } } peers { permanent_uuid: "02ab510d1c094169b31adb687a154454" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42469 } }
I20260925 16:32:56.434875 2014 raft_consensus.cc:374] T 0d5ad44c65da4013a7ea47dc61d7381f P f1e3c666b63f43c796658fd44d19244e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:32:56.434957 2014 raft_consensus.cc:729] T 0d5ad44c65da4013a7ea47dc61d7381f P f1e3c666b63f43c796658fd44d19244e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e3c666b63f43c796658fd44d19244e, State: Initialized, Role: FOLLOWER
I20260925 16:32:56.434859 2015 raft_consensus.cc:348] T 0d5ad44c65da4013a7ea47dc61d7381f P 02ab510d1c094169b31adb687a154454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e07d066bb864504b44c19f75dc565b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44121 } } peers { permanent_uuid: "f1e3c666b63f43c796658fd44d19244e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46439 } } peers { permanent_uuid: "02ab510d1c094169b31adb687a154454" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42469 } }
I20260925 16:32:56.435105 2015 raft_consensus.cc:374] T 0d5ad44c65da4013a7ea47dc61d7381f P 02ab510d1c094169b31adb687a154454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:32:56.435161 2015 raft_consensus.cc:729] T 0d5ad44c65da4013a7ea47dc61d7381f P 02ab510d1c094169b31adb687a154454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02ab510d1c094169b31adb687a154454, State: Initialized, Role: FOLLOWER
I20260925 16:32:56.435463 2014 consensus_queue.cc:260] T 0d5ad44c65da4013a7ea47dc61d7381f P f1e3c666b63f43c796658fd44d19244e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e07d066bb864504b44c19f75dc565b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44121 } } peers { permanent_uuid: "f1e3c666b63f43c796658fd44d19244e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46439 } } peers { permanent_uuid: "02ab510d1c094169b31adb687a154454" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42469 } }
I20260925 16:32:56.435576 2015 consensus_queue.cc:260] T 0d5ad44c65da4013a7ea47dc61d7381f P 02ab510d1c094169b31adb687a154454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e07d066bb864504b44c19f75dc565b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44121 } } peers { permanent_uuid: "f1e3c666b63f43c796658fd44d19244e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46439 } } peers { permanent_uuid: "02ab510d1c094169b31adb687a154454" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42469 } }
I20260925 16:32:56.436074 1850 heartbeater.cc:503] Master 127.0.98.62:45269 was elected leader, sending a full tablet report...
I20260925 16:32:56.436419 2014 ts_tablet_manager.cc:1428] T 0d5ad44c65da4013a7ea47dc61d7381f P f1e3c666b63f43c796658fd44d19244e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 16:32:56.436491 2015 ts_tablet_manager.cc:1428] T 0d5ad44c65da4013a7ea47dc61d7381f P 02ab510d1c094169b31adb687a154454: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20260925 16:32:56.436805 1990 heartbeater.cc:503] Master 127.0.98.62:45269 was elected leader, sending a full tablet report...
I20260925 16:32:56.559347 2021 raft_consensus.cc:482] T 0d5ad44c65da4013a7ea47dc61d7381f P 02ab510d1c094169b31adb687a154454 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:32:56.559552 2021 raft_consensus.cc:504] T 0d5ad44c65da4013a7ea47dc61d7381f P 02ab510d1c094169b31adb687a154454 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e07d066bb864504b44c19f75dc565b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44121 } } peers { permanent_uuid: "f1e3c666b63f43c796658fd44d19244e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46439 } } peers { permanent_uuid: "02ab510d1c094169b31adb687a154454" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42469 } }
I20260925 16:32:56.560814 2021 leader_election.cc:288] T 0d5ad44c65da4013a7ea47dc61d7381f P 02ab510d1c094169b31adb687a154454 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e07d066bb864504b44c19f75dc565b7 (127.0.98.1:44121), f1e3c666b63f43c796658fd44d19244e (127.0.98.2:46439)
I20260925 16:32:56.565317 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:56.561289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44299 (local address 127.0.98.1:44121)
0925 16:32:56.561426 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:56.561431 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:56.561446 (+ 15us) server_negotiation.cc:407] Connection header received
0925 16:32:56.561566 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:56.561570 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:56.561622 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:56.561709 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:32:56.561978 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:56.562721 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:56.564273 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:56.564690 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:56.564820 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:56.564893 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:56.565090 (+ 197us) server_negotiation.cc:299] Negotiation successful
0925 16:32:56.565160 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 16:32:56.565337 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:56.561093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44121 (local address 127.0.98.3:44299)
0925 16:32:56.561327 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:56.561347 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 16:32:56.561449 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:32:56.561717 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:32:56.561720 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:32:56.561735 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:32:56.561862 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:56.561867 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:56.562850 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:56.562855 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:56.564113 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:56.564123 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:56.564833 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:56.564837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:56.564942 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:56.564974 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 16:32:56.565074 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 16:32:56.565159 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137}
I20260925 16:32:56.565521 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:56.561233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46171 (local address 127.0.98.2:46439)
0925 16:32:56.561400 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:56.561406 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:56.562037 (+ 631us) server_negotiation.cc:407] Connection header received
0925 16:32:56.562216 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:56.562219 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:56.562275 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:56.562369 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:32:56.562701 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:56.563382 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:56.564454 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:56.564840 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:56.565028 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:56.565089 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:56.565322 (+ 233us) server_negotiation.cc:299] Negotiation successful
0925 16:32:56.565396 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 16:32:56.565546 2022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:56.561093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46439 (local address 127.0.98.3:46171)
0925 16:32:56.561758 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:56.561771 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:32:56.562080 (+ 309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:32:56.562388 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:32:56.562392 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:32:56.562408 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:32:56.562537 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:56.562543 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:56.563527 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:56.563533 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:56.564315 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:56.564323 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:56.565023 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:56.565026 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:56.565129 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:56.565154 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 16:32:56.565308 (+ 154us) client_negotiation.cc:240] Negotiation successful
0925 16:32:56.565400 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"mutex_wait_us":259,"thread_start_us":181,"threads_started":1}
I20260925 16:32:56.566183 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d5ad44c65da4013a7ea47dc61d7381f" candidate_uuid: "02ab510d1c094169b31adb687a154454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e3c666b63f43c796658fd44d19244e" is_pre_election: true
I20260925 16:32:56.566187 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d5ad44c65da4013a7ea47dc61d7381f" candidate_uuid: "02ab510d1c094169b31adb687a154454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e07d066bb864504b44c19f75dc565b7" is_pre_election: true
I20260925 16:32:56.566484 1804 raft_consensus.cc:2454] T 0d5ad44c65da4013a7ea47dc61d7381f P f1e3c666b63f43c796658fd44d19244e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02ab510d1c094169b31adb687a154454 in term 0.
I20260925 16:32:56.566586 1664 raft_consensus.cc:2454] T 0d5ad44c65da4013a7ea47dc61d7381f P 5e07d066bb864504b44c19f75dc565b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02ab510d1c094169b31adb687a154454 in term 0.
I20260925 16:32:56.567068 1875 leader_election.cc:302] T 0d5ad44c65da4013a7ea47dc61d7381f P 02ab510d1c094169b31adb687a154454 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02ab510d1c094169b31adb687a154454, f1e3c666b63f43c796658fd44d19244e; no voters:
I20260925 16:32:56.567353 2021 raft_consensus.cc:2781] T 0d5ad44c65da4013a7ea47dc61d7381f P 02ab510d1c094169b31adb687a154454 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:32:56.567430 2021 raft_consensus.cc:482] T 0d5ad44c65da4013a7ea47dc61d7381f P 02ab510d1c094169b31adb687a154454 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:32:56.567461 2021 raft_consensus.cc:3037] T 0d5ad44c65da4013a7ea47dc61d7381f P 02ab510d1c094169b31adb687a154454 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:32:56.568545 2021 raft_consensus.cc:504] T 0d5ad44c65da4013a7ea47dc61d7381f P 02ab510d1c094169b31adb687a154454 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e07d066bb864504b44c19f75dc565b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44121 } } peers { permanent_uuid: "f1e3c666b63f43c796658fd44d19244e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46439 } } peers { permanent_uuid: "02ab510d1c094169b31adb687a154454" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42469 } }
I20260925 16:32:56.569000 2021 leader_election.cc:288] T 0d5ad44c65da4013a7ea47dc61d7381f P 02ab510d1c094169b31adb687a154454 [CANDIDATE]: Term 1 election: Requested vote from peers 5e07d066bb864504b44c19f75dc565b7 (127.0.98.1:44121), f1e3c666b63f43c796658fd44d19244e (127.0.98.2:46439)
I20260925 16:32:56.569416 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d5ad44c65da4013a7ea47dc61d7381f" candidate_uuid: "02ab510d1c094169b31adb687a154454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e3c666b63f43c796658fd44d19244e"
I20260925 16:32:56.569416 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d5ad44c65da4013a7ea47dc61d7381f" candidate_uuid: "02ab510d1c094169b31adb687a154454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e07d066bb864504b44c19f75dc565b7"
I20260925 16:32:56.569554 1664 raft_consensus.cc:3037] T 0d5ad44c65da4013a7ea47dc61d7381f P 5e07d066bb864504b44c19f75dc565b7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:32:56.569554 1804 raft_consensus.cc:3037] T 0d5ad44c65da4013a7ea47dc61d7381f P f1e3c666b63f43c796658fd44d19244e [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:32:56.571063 1664 raft_consensus.cc:2454] T 0d5ad44c65da4013a7ea47dc61d7381f P 5e07d066bb864504b44c19f75dc565b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02ab510d1c094169b31adb687a154454 in term 1.
I20260925 16:32:56.571445 1877 leader_election.cc:302] T 0d5ad44c65da4013a7ea47dc61d7381f P 02ab510d1c094169b31adb687a154454 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02ab510d1c094169b31adb687a154454, 5e07d066bb864504b44c19f75dc565b7; no voters:
I20260925 16:32:56.571604 1804 raft_consensus.cc:2454] T 0d5ad44c65da4013a7ea47dc61d7381f P f1e3c666b63f43c796658fd44d19244e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02ab510d1c094169b31adb687a154454 in term 1.
I20260925 16:32:56.571812 2021 raft_consensus.cc:2781] T 0d5ad44c65da4013a7ea47dc61d7381f P 02ab510d1c094169b31adb687a154454 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:32:56.572072 2021 raft_consensus.cc:686] T 0d5ad44c65da4013a7ea47dc61d7381f P 02ab510d1c094169b31adb687a154454 [term 1 LEADER]: Becoming Leader. State: Replica: 02ab510d1c094169b31adb687a154454, State: Running, Role: LEADER
I20260925 16:32:56.572434 2021 consensus_queue.cc:237] T 0d5ad44c65da4013a7ea47dc61d7381f P 02ab510d1c094169b31adb687a154454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e07d066bb864504b44c19f75dc565b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44121 } } peers { permanent_uuid: "f1e3c666b63f43c796658fd44d19244e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46439 } } peers { permanent_uuid: "02ab510d1c094169b31adb687a154454" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42469 } }
I20260925 16:32:56.575834 1505 catalog_manager.cc:5495] T 0d5ad44c65da4013a7ea47dc61d7381f P 02ab510d1c094169b31adb687a154454 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02ab510d1c094169b31adb687a154454 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "02ab510d1c094169b31adb687a154454" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e07d066bb864504b44c19f75dc565b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1e3c666b63f43c796658fd44d19244e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02ab510d1c094169b31adb687a154454" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42469 } health_report { overall_health: HEALTHY } } }
W20260925 16:32:56.606995 1991 tablet.cc:2369] T 0d5ad44c65da4013a7ea47dc61d7381f P 02ab510d1c094169b31adb687a154454: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:32:56.608745 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:56.605037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55084 (local address 127.0.98.3:42469)
0925 16:32:56.605159 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:56.605162 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:56.605177 (+ 15us) server_negotiation.cc:407] Connection header received
0925 16:32:56.605358 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:56.605360 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:56.605414 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:56.605506 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:32:56.605731 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:56.606225 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:56.607242 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:56.607471 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:56.607549 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:56.607863 (+ 314us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:32:56.608340 (+ 477us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:32:56.608464 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:56.608526 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 16:32:56.608576 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 16:32:56.614411 1664 raft_consensus.cc:1261] T 0d5ad44c65da4013a7ea47dc61d7381f P 5e07d066bb864504b44c19f75dc565b7 [term 1 FOLLOWER]: Refusing update from remote peer 02ab510d1c094169b31adb687a154454: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:32:56.614429 1804 raft_consensus.cc:1261] T 0d5ad44c65da4013a7ea47dc61d7381f P f1e3c666b63f43c796658fd44d19244e [term 1 FOLLOWER]: Refusing update from remote peer 02ab510d1c094169b31adb687a154454: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:32:56.615181 2023 consensus_queue.cc:1035] T 0d5ad44c65da4013a7ea47dc61d7381f P 02ab510d1c094169b31adb687a154454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e07d066bb864504b44c19f75dc565b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:32:56.615357 2021 consensus_queue.cc:1035] T 0d5ad44c65da4013a7ea47dc61d7381f P 02ab510d1c094169b31adb687a154454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e3c666b63f43c796658fd44d19244e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:32:56.624311 2028 mvcc.cc:205] Tried to move back new op lower bound from 7333289888205590528 to 7333289888043687936. Current Snapshot: MvccSnapshot[applied={T|T < 7333289888205590528}]
I20260925 16:32:56.626490 2027 mvcc.cc:205] Tried to move back new op lower bound from 7333289888205590528 to 7333289888043687936. Current Snapshot: MvccSnapshot[applied={T|T < 7333289888205590528}]
I20260925 16:32:56.635982 1507 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:32:56.636178 1507 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:32:56.638891 1507 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eec9855ba8004aecae4b7c47ecce51d3: Sending DeleteTablet for 3 replicas of tablet 0d5ad44c65da4013a7ea47dc61d7381f
I20260925 16:32:56.639853 1784 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d5ad44c65da4013a7ea47dc61d7381f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:32:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56058
I20260925 16:32:56.640023 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d5ad44c65da4013a7ea47dc61d7381f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:32:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55070
I20260925 16:32:56.640027 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d5ad44c65da4013a7ea47dc61d7381f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:32:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47804
I20260925 16:32:56.640363 2035 tablet_replica.cc:330] T 0d5ad44c65da4013a7ea47dc61d7381f P f1e3c666b63f43c796658fd44d19244e: stopping tablet replica
I20260925 16:32:56.640663 2035 raft_consensus.cc:2229] T 0d5ad44c65da4013a7ea47dc61d7381f P f1e3c666b63f43c796658fd44d19244e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:32:56.640974 2035 raft_consensus.cc:2258] T 0d5ad44c65da4013a7ea47dc61d7381f P f1e3c666b63f43c796658fd44d19244e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:32:56.641273 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52950:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 16:32:56.641726 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:32:56.642516 2035 ts_tablet_manager.cc:1905] T 0d5ad44c65da4013a7ea47dc61d7381f P f1e3c666b63f43c796658fd44d19244e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:32:56.645329 2035 ts_tablet_manager.cc:1918] T 0d5ad44c65da4013a7ea47dc61d7381f P f1e3c666b63f43c796658fd44d19244e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:32:56.645427 2035 log.cc:1196] T 0d5ad44c65da4013a7ea47dc61d7381f P f1e3c666b63f43c796658fd44d19244e: Deleting WAL directory at /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/minicluster-data/ts-1/wal/wals/0d5ad44c65da4013a7ea47dc61d7381f
I20260925 16:32:56.645736 2035 ts_tablet_manager.cc:1939] T 0d5ad44c65da4013a7ea47dc61d7381f P f1e3c666b63f43c796658fd44d19244e: Deleting consensus metadata
I20260925 16:32:56.646459 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 9c984fdf71cc471386fffea9daeab895 (DEFAULT_TABLE table=test-table [id=72b11d68f4f54d96814aa4c2faf10ea5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:32:56.646524 1784 tablet_service.cc:1467] Processing CreateTablet for tablet 9c984fdf71cc471386fffea9daeab895 (DEFAULT_TABLE table=test-table [id=72b11d68f4f54d96814aa4c2faf10ea5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:32:56.646778 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c984fdf71cc471386fffea9daeab895. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:32:56.646819 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c984fdf71cc471386fffea9daeab895. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:32:56.648691 2013 tablet_bootstrap.cc:492] T 9c984fdf71cc471386fffea9daeab895 P 5e07d066bb864504b44c19f75dc565b7: Bootstrap starting.
I20260925 16:32:56.648984 2014 tablet_bootstrap.cc:492] T 9c984fdf71cc471386fffea9daeab895 P f1e3c666b63f43c796658fd44d19244e: Bootstrap starting.
I20260925 16:32:56.649613 2013 tablet_bootstrap.cc:654] T 9c984fdf71cc471386fffea9daeab895 P 5e07d066bb864504b44c19f75dc565b7: Neither blocks nor log segments found. Creating new log.
I20260925 16:32:56.649865 2014 tablet_bootstrap.cc:654] T 9c984fdf71cc471386fffea9daeab895 P f1e3c666b63f43c796658fd44d19244e: Neither blocks nor log segments found. Creating new log.
I20260925 16:32:56.650414 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 9c984fdf71cc471386fffea9daeab895 (DEFAULT_TABLE table=test-table [id=72b11d68f4f54d96814aa4c2faf10ea5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:32:56.650702 2037 tablet_replica.cc:330] T 0d5ad44c65da4013a7ea47dc61d7381f P 5e07d066bb864504b44c19f75dc565b7: stopping tablet replica
I20260925 16:32:56.650712 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c984fdf71cc471386fffea9daeab895. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:32:56.651015 2037 raft_consensus.cc:2229] T 0d5ad44c65da4013a7ea47dc61d7381f P 5e07d066bb864504b44c19f75dc565b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:32:56.651301 2037 raft_consensus.cc:2258] T 0d5ad44c65da4013a7ea47dc61d7381f P 5e07d066bb864504b44c19f75dc565b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:32:56.652175 1491 catalog_manager.cc:4841] TS f1e3c666b63f43c796658fd44d19244e (127.0.98.2:46439): tablet 0d5ad44c65da4013a7ea47dc61d7381f (table test-table [id=d569bf9030864380b5e4179b249e4b43]) successfully deleted
I20260925 16:32:56.652657 2014 tablet_bootstrap.cc:492] T 9c984fdf71cc471386fffea9daeab895 P f1e3c666b63f43c796658fd44d19244e: No bootstrap required, opened a new log
I20260925 16:32:56.652725 2014 ts_tablet_manager.cc:1397] T 9c984fdf71cc471386fffea9daeab895 P f1e3c666b63f43c796658fd44d19244e: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260925 16:32:56.653115 2014 raft_consensus.cc:348] T 9c984fdf71cc471386fffea9daeab895 P f1e3c666b63f43c796658fd44d19244e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e07d066bb864504b44c19f75dc565b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44121 } } peers { permanent_uuid: "f1e3c666b63f43c796658fd44d19244e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46439 } } peers { permanent_uuid: "02ab510d1c094169b31adb687a154454" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42469 } }
I20260925 16:32:56.653195 2015 tablet_bootstrap.cc:492] T 9c984fdf71cc471386fffea9daeab895 P 02ab510d1c094169b31adb687a154454: Bootstrap starting.
I20260925 16:32:56.653223 2014 raft_consensus.cc:374] T 9c984fdf71cc471386fffea9daeab895 P f1e3c666b63f43c796658fd44d19244e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:32:56.653254 2014 raft_consensus.cc:729] T 9c984fdf71cc471386fffea9daeab895 P f1e3c666b63f43c796658fd44d19244e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e3c666b63f43c796658fd44d19244e, State: Initialized, Role: FOLLOWER
I20260925 16:32:56.653366 2014 consensus_queue.cc:260] T 9c984fdf71cc471386fffea9daeab895 P f1e3c666b63f43c796658fd44d19244e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e07d066bb864504b44c19f75dc565b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44121 } } peers { permanent_uuid: "f1e3c666b63f43c796658fd44d19244e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46439 } } peers { permanent_uuid: "02ab510d1c094169b31adb687a154454" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42469 } }
I20260925 16:32:56.653595 2014 ts_tablet_manager.cc:1428] T 9c984fdf71cc471386fffea9daeab895 P f1e3c666b63f43c796658fd44d19244e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 16:32:56.654628 2015 tablet_bootstrap.cc:654] T 9c984fdf71cc471386fffea9daeab895 P 02ab510d1c094169b31adb687a154454: Neither blocks nor log segments found. Creating new log.
I20260925 16:32:56.655182 2013 tablet_bootstrap.cc:492] T 9c984fdf71cc471386fffea9daeab895 P 5e07d066bb864504b44c19f75dc565b7: No bootstrap required, opened a new log
I20260925 16:32:56.655268 2013 ts_tablet_manager.cc:1397] T 9c984fdf71cc471386fffea9daeab895 P 5e07d066bb864504b44c19f75dc565b7: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260925 16:32:56.655422 2036 tablet_replica.cc:330] T 0d5ad44c65da4013a7ea47dc61d7381f P 02ab510d1c094169b31adb687a154454: stopping tablet replica
I20260925 16:32:56.655726 2036 raft_consensus.cc:2229] T 0d5ad44c65da4013a7ea47dc61d7381f P 02ab510d1c094169b31adb687a154454 [term 1 LEADER]: Raft consensus shutting down.
I20260925 16:32:56.655757 2013 raft_consensus.cc:348] T 9c984fdf71cc471386fffea9daeab895 P 5e07d066bb864504b44c19f75dc565b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e07d066bb864504b44c19f75dc565b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44121 } } peers { permanent_uuid: "f1e3c666b63f43c796658fd44d19244e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46439 } } peers { permanent_uuid: "02ab510d1c094169b31adb687a154454" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42469 } }
I20260925 16:32:56.655885 2013 raft_consensus.cc:374] T 9c984fdf71cc471386fffea9daeab895 P 5e07d066bb864504b44c19f75dc565b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:32:56.655922 2013 raft_consensus.cc:729] T 9c984fdf71cc471386fffea9daeab895 P 5e07d066bb864504b44c19f75dc565b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e07d066bb864504b44c19f75dc565b7, State: Initialized, Role: FOLLOWER
I20260925 16:32:56.656064 2036 raft_consensus.cc:2258] T 0d5ad44c65da4013a7ea47dc61d7381f P 02ab510d1c094169b31adb687a154454 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:32:56.656040 2013 consensus_queue.cc:260] T 9c984fdf71cc471386fffea9daeab895 P 5e07d066bb864504b44c19f75dc565b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e07d066bb864504b44c19f75dc565b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44121 } } peers { permanent_uuid: "f1e3c666b63f43c796658fd44d19244e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46439 } } peers { permanent_uuid: "02ab510d1c094169b31adb687a154454" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42469 } }
I20260925 16:32:56.656283 2013 ts_tablet_manager.cc:1428] T 9c984fdf71cc471386fffea9daeab895 P 5e07d066bb864504b44c19f75dc565b7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 16:32:56.656869 2037 ts_tablet_manager.cc:1905] T 0d5ad44c65da4013a7ea47dc61d7381f P 5e07d066bb864504b44c19f75dc565b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:32:56.657653 2036 ts_tablet_manager.cc:1905] T 0d5ad44c65da4013a7ea47dc61d7381f P 02ab510d1c094169b31adb687a154454: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:32:56.660436 2036 ts_tablet_manager.cc:1918] T 0d5ad44c65da4013a7ea47dc61d7381f P 02ab510d1c094169b31adb687a154454: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:32:56.660537 2036 log.cc:1196] T 0d5ad44c65da4013a7ea47dc61d7381f P 02ab510d1c094169b31adb687a154454: Deleting WAL directory at /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/minicluster-data/ts-2/wal/wals/0d5ad44c65da4013a7ea47dc61d7381f
I20260925 16:32:56.660537 2037 ts_tablet_manager.cc:1918] T 0d5ad44c65da4013a7ea47dc61d7381f P 5e07d066bb864504b44c19f75dc565b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:32:56.660629 2037 log.cc:1196] T 0d5ad44c65da4013a7ea47dc61d7381f P 5e07d066bb864504b44c19f75dc565b7: Deleting WAL directory at /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/minicluster-data/ts-0/wal/wals/0d5ad44c65da4013a7ea47dc61d7381f
I20260925 16:32:56.660866 2036 ts_tablet_manager.cc:1939] T 0d5ad44c65da4013a7ea47dc61d7381f P 02ab510d1c094169b31adb687a154454: Deleting consensus metadata
I20260925 16:32:56.660996 2037 ts_tablet_manager.cc:1939] T 0d5ad44c65da4013a7ea47dc61d7381f P 5e07d066bb864504b44c19f75dc565b7: Deleting consensus metadata
I20260925 16:32:56.662087 1493 catalog_manager.cc:4841] TS 5e07d066bb864504b44c19f75dc565b7 (127.0.98.1:44121): tablet 0d5ad44c65da4013a7ea47dc61d7381f (table test-table [id=d569bf9030864380b5e4179b249e4b43]) successfully deleted
I20260925 16:32:56.662250 1493 catalog_manager.cc:4841] TS 02ab510d1c094169b31adb687a154454 (127.0.98.3:42469): tablet 0d5ad44c65da4013a7ea47dc61d7381f (table test-table [id=d569bf9030864380b5e4179b249e4b43]) successfully deleted
I20260925 16:32:56.662709 2015 tablet_bootstrap.cc:492] T 9c984fdf71cc471386fffea9daeab895 P 02ab510d1c094169b31adb687a154454: No bootstrap required, opened a new log
I20260925 16:32:56.662825 2015 ts_tablet_manager.cc:1397] T 9c984fdf71cc471386fffea9daeab895 P 02ab510d1c094169b31adb687a154454: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.001s
I20260925 16:32:56.663316 2015 raft_consensus.cc:348] T 9c984fdf71cc471386fffea9daeab895 P 02ab510d1c094169b31adb687a154454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e07d066bb864504b44c19f75dc565b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44121 } } peers { permanent_uuid: "f1e3c666b63f43c796658fd44d19244e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46439 } } peers { permanent_uuid: "02ab510d1c094169b31adb687a154454" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42469 } }
I20260925 16:32:56.663430 2015 raft_consensus.cc:374] T 9c984fdf71cc471386fffea9daeab895 P 02ab510d1c094169b31adb687a154454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:32:56.663470 2015 raft_consensus.cc:729] T 9c984fdf71cc471386fffea9daeab895 P 02ab510d1c094169b31adb687a154454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02ab510d1c094169b31adb687a154454, State: Initialized, Role: FOLLOWER
I20260925 16:32:56.663609 2015 consensus_queue.cc:260] T 9c984fdf71cc471386fffea9daeab895 P 02ab510d1c094169b31adb687a154454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e07d066bb864504b44c19f75dc565b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44121 } } peers { permanent_uuid: "f1e3c666b63f43c796658fd44d19244e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46439 } } peers { permanent_uuid: "02ab510d1c094169b31adb687a154454" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42469 } }
I20260925 16:32:56.663883 2015 ts_tablet_manager.cc:1428] T 9c984fdf71cc471386fffea9daeab895 P 02ab510d1c094169b31adb687a154454: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20260925 16:32:56.889320 1851 tablet.cc:2369] T 9c984fdf71cc471386fffea9daeab895 P f1e3c666b63f43c796658fd44d19244e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 16:32:56.899698 1711 tablet.cc:2369] T 9c984fdf71cc471386fffea9daeab895 P 5e07d066bb864504b44c19f75dc565b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:32:56.921455 2019 raft_consensus.cc:482] T 9c984fdf71cc471386fffea9daeab895 P 5e07d066bb864504b44c19f75dc565b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:32:56.921603 2019 raft_consensus.cc:504] T 9c984fdf71cc471386fffea9daeab895 P 5e07d066bb864504b44c19f75dc565b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e07d066bb864504b44c19f75dc565b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44121 } } peers { permanent_uuid: "f1e3c666b63f43c796658fd44d19244e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46439 } } peers { permanent_uuid: "02ab510d1c094169b31adb687a154454" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42469 } }
I20260925 16:32:56.922705 2019 leader_election.cc:288] T 9c984fdf71cc471386fffea9daeab895 P 5e07d066bb864504b44c19f75dc565b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e3c666b63f43c796658fd44d19244e (127.0.98.2:46439), 02ab510d1c094169b31adb687a154454 (127.0.98.3:42469)
I20260925 16:32:56.926689 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:56.923108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35715 (local address 127.0.98.3:42469)
0925 16:32:56.923239 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:56.923245 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:56.923449 (+ 204us) server_negotiation.cc:407] Connection header received
0925 16:32:56.923591 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:56.923594 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:56.923632 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:56.923720 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:32:56.924018 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:56.924502 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:56.925720 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:56.926105 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:56.926227 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:56.926296 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:56.926496 (+ 200us) server_negotiation.cc:299] Negotiation successful
0925 16:32:56.926564 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 16:32:56.926703 2041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:56.922897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46439 (local address 127.0.98.1:55149)
0925 16:32:56.923231 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:56.923253 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 16:32:56.923411 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:32:56.923721 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:32:56.923724 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:32:56.923739 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:32:56.923873 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:56.923879 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:56.924655 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:56.924659 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:56.925571 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:56.925580 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:56.926227 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:56.926230 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:56.926323 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:56.926344 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 16:32:56.926420 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 16:32:56.926511 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"mutex_wait_us":57,"spinlock_wait_cycles":10496,"thread_start_us":75,"threads_started":1}
I20260925 16:32:56.926713 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:56.923028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55149 (local address 127.0.98.2:46439)
0925 16:32:56.923175 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:56.923180 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:56.923346 (+ 166us) server_negotiation.cc:407] Connection header received
0925 16:32:56.923563 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:56.923569 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:56.923627 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:56.923684 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:32:56.923984 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:56.924505 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:56.925690 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:56.926098 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:56.926234 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:56.926297 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:56.926456 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 16:32:56.926528 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 16:32:56.926934 2042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:56.923020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42469 (local address 127.0.98.1:35715)
0925 16:32:56.923363 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:56.923374 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 16:32:56.923464 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:32:56.923748 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:32:56.923751 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:32:56.923795 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:32:56.923892 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:56.923897 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:56.924617 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:56.924623 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:56.925567 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:56.925577 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:56.926263 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:56.926267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:56.926357 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:56.926373 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 16:32:56.926467 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 16:32:56.926530 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":41,"spinlock_wait_cycles":3840,"thread_start_us":73,"threads_started":1}
I20260925 16:32:56.927198 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c984fdf71cc471386fffea9daeab895" candidate_uuid: "5e07d066bb864504b44c19f75dc565b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e3c666b63f43c796658fd44d19244e" is_pre_election: true
I20260925 16:32:56.927366 1804 raft_consensus.cc:2454] T 9c984fdf71cc471386fffea9daeab895 P f1e3c666b63f43c796658fd44d19244e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e07d066bb864504b44c19f75dc565b7 in term 0.
I20260925 16:32:56.927522 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c984fdf71cc471386fffea9daeab895" candidate_uuid: "5e07d066bb864504b44c19f75dc565b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02ab510d1c094169b31adb687a154454" is_pre_election: true
I20260925 16:32:56.927778 1944 raft_consensus.cc:2454] T 9c984fdf71cc471386fffea9daeab895 P 02ab510d1c094169b31adb687a154454 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e07d066bb864504b44c19f75dc565b7 in term 0.
I20260925 16:32:56.927874 1595 leader_election.cc:302] T 9c984fdf71cc471386fffea9daeab895 P 5e07d066bb864504b44c19f75dc565b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e07d066bb864504b44c19f75dc565b7, f1e3c666b63f43c796658fd44d19244e; no voters:
I20260925 16:32:56.928189 2019 raft_consensus.cc:2781] T 9c984fdf71cc471386fffea9daeab895 P 5e07d066bb864504b44c19f75dc565b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:32:56.928259 2019 raft_consensus.cc:482] T 9c984fdf71cc471386fffea9daeab895 P 5e07d066bb864504b44c19f75dc565b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:32:56.928313 2019 raft_consensus.cc:3037] T 9c984fdf71cc471386fffea9daeab895 P 5e07d066bb864504b44c19f75dc565b7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:32:56.929411 2019 raft_consensus.cc:504] T 9c984fdf71cc471386fffea9daeab895 P 5e07d066bb864504b44c19f75dc565b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e07d066bb864504b44c19f75dc565b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44121 } } peers { permanent_uuid: "f1e3c666b63f43c796658fd44d19244e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46439 } } peers { permanent_uuid: "02ab510d1c094169b31adb687a154454" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42469 } }
I20260925 16:32:56.929745 2019 leader_election.cc:288] T 9c984fdf71cc471386fffea9daeab895 P 5e07d066bb864504b44c19f75dc565b7 [CANDIDATE]: Term 1 election: Requested vote from peers f1e3c666b63f43c796658fd44d19244e (127.0.98.2:46439), 02ab510d1c094169b31adb687a154454 (127.0.98.3:42469)
I20260925 16:32:56.930109 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c984fdf71cc471386fffea9daeab895" candidate_uuid: "5e07d066bb864504b44c19f75dc565b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e3c666b63f43c796658fd44d19244e"
I20260925 16:32:56.930209 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c984fdf71cc471386fffea9daeab895" candidate_uuid: "5e07d066bb864504b44c19f75dc565b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02ab510d1c094169b31adb687a154454"
I20260925 16:32:56.930250 1804 raft_consensus.cc:3037] T 9c984fdf71cc471386fffea9daeab895 P f1e3c666b63f43c796658fd44d19244e [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:32:56.930336 1944 raft_consensus.cc:3037] T 9c984fdf71cc471386fffea9daeab895 P 02ab510d1c094169b31adb687a154454 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:32:56.931444 1804 raft_consensus.cc:2454] T 9c984fdf71cc471386fffea9daeab895 P f1e3c666b63f43c796658fd44d19244e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e07d066bb864504b44c19f75dc565b7 in term 1.
I20260925 16:32:56.931444 1944 raft_consensus.cc:2454] T 9c984fdf71cc471386fffea9daeab895 P 02ab510d1c094169b31adb687a154454 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e07d066bb864504b44c19f75dc565b7 in term 1.
I20260925 16:32:56.931715 1595 leader_election.cc:302] T 9c984fdf71cc471386fffea9daeab895 P 5e07d066bb864504b44c19f75dc565b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e07d066bb864504b44c19f75dc565b7, f1e3c666b63f43c796658fd44d19244e; no voters:
I20260925 16:32:56.931914 2019 raft_consensus.cc:2781] T 9c984fdf71cc471386fffea9daeab895 P 5e07d066bb864504b44c19f75dc565b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:32:56.932128 2019 raft_consensus.cc:686] T 9c984fdf71cc471386fffea9daeab895 P 5e07d066bb864504b44c19f75dc565b7 [term 1 LEADER]: Becoming Leader. State: Replica: 5e07d066bb864504b44c19f75dc565b7, State: Running, Role: LEADER
I20260925 16:32:56.932381 2019 consensus_queue.cc:237] T 9c984fdf71cc471386fffea9daeab895 P 5e07d066bb864504b44c19f75dc565b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e07d066bb864504b44c19f75dc565b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44121 } } peers { permanent_uuid: "f1e3c666b63f43c796658fd44d19244e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46439 } } peers { permanent_uuid: "02ab510d1c094169b31adb687a154454" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42469 } }
I20260925 16:32:56.934648 1507 catalog_manager.cc:5495] T 9c984fdf71cc471386fffea9daeab895 P 5e07d066bb864504b44c19f75dc565b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e07d066bb864504b44c19f75dc565b7 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "5e07d066bb864504b44c19f75dc565b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e07d066bb864504b44c19f75dc565b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44121 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1e3c666b63f43c796658fd44d19244e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02ab510d1c094169b31adb687a154454" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42469 } health_report { overall_health: UNKNOWN } } }
Sep 25 16:32:56 dist-test-slave-65pf krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:32:56.979543 2050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:56.964301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45269 (local address 127.0.0.1:52960)
0925 16:32:56.964605 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:56.964618 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:32:56.964712 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:32:56.964984 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:32:56.964986 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:32:56.965370 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:32:56.965471 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:56.965477 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:56.966182 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:56.966184 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:56.966797 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:56.966805 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:56.967191 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:56.967194 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:56.967252 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:56.974849 (+ 7597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:32:56.974862 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:32:56.976364 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:32:56.978063 (+ 1699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:56.978072 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:56.978077 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:56.978299 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:56.978554 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:56.978557 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:56.978559 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:56.978687 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:56.979091 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:32:56.979097 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:32:56.979202 (+ 105us) client_negotiation.cc:769] Sending connection context
0925 16:32:56.979284 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 16:32:56.979399 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":125,"threads_started":1}
I20260925 16:32:56.979625 2051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:56.964337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52960 (local address 127.0.98.62:45269)
0925 16:32:56.964691 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:56.964698 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:56.964712 (+ 14us) server_negotiation.cc:407] Connection header received
0925 16:32:56.964807 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:56.964809 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:56.964847 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:56.964948 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:56.965566 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:56.966095 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:56.966897 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:56.967104 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:56.967178 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:56.976530 (+ 9352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:32:56.976534 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:32:56.976536 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:32:56.976557 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:32:56.977970 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:56.978389 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:56.978392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:56.978393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:56.978464 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:56.978774 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:56.978777 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:56.978778 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:56.979004 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:32:56.979080 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:56.979365 (+ 285us) server_negotiation.cc:299] Negotiation successful
0925 16:32:56.979476 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":104,"threads_started":1}
I20260925 16:32:56.983425 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52960:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 16:32:56.984143 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:32:56.988874 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 72d5dd9016634abf837c6661d977499e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=67bc44aa212746128790a9e56c6677ef]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 16:32:56.989051 1784 tablet_service.cc:1467] Processing CreateTablet for tablet 72d5dd9016634abf837c6661d977499e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=67bc44aa212746128790a9e56c6677ef]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 16:32:56.989319 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72d5dd9016634abf837c6661d977499e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:32:56.989419 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72d5dd9016634abf837c6661d977499e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:32:56.989387 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 72d5dd9016634abf837c6661d977499e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=67bc44aa212746128790a9e56c6677ef]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 16:32:56.989724 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72d5dd9016634abf837c6661d977499e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:32:56.992085 2013 tablet_bootstrap.cc:492] T 72d5dd9016634abf837c6661d977499e P 5e07d066bb864504b44c19f75dc565b7: Bootstrap starting.
I20260925 16:32:56.992295 2014 tablet_bootstrap.cc:492] T 72d5dd9016634abf837c6661d977499e P f1e3c666b63f43c796658fd44d19244e: Bootstrap starting.
I20260925 16:32:56.992455 2015 tablet_bootstrap.cc:492] T 72d5dd9016634abf837c6661d977499e P 02ab510d1c094169b31adb687a154454: Bootstrap starting.
I20260925 16:32:56.993011 2013 tablet_bootstrap.cc:654] T 72d5dd9016634abf837c6661d977499e P 5e07d066bb864504b44c19f75dc565b7: Neither blocks nor log segments found. Creating new log.
I20260925 16:32:56.993271 2015 tablet_bootstrap.cc:654] T 72d5dd9016634abf837c6661d977499e P 02ab510d1c094169b31adb687a154454: Neither blocks nor log segments found. Creating new log.
I20260925 16:32:56.993264 2014 tablet_bootstrap.cc:654] T 72d5dd9016634abf837c6661d977499e P f1e3c666b63f43c796658fd44d19244e: Neither blocks nor log segments found. Creating new log.
I20260925 16:32:56.994500 2015 tablet_bootstrap.cc:492] T 72d5dd9016634abf837c6661d977499e P 02ab510d1c094169b31adb687a154454: No bootstrap required, opened a new log
I20260925 16:32:56.994623 2015 ts_tablet_manager.cc:1397] T 72d5dd9016634abf837c6661d977499e P 02ab510d1c094169b31adb687a154454: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 16:32:56.994789 2013 tablet_bootstrap.cc:492] T 72d5dd9016634abf837c6661d977499e P 5e07d066bb864504b44c19f75dc565b7: No bootstrap required, opened a new log
I20260925 16:32:56.994848 2013 ts_tablet_manager.cc:1397] T 72d5dd9016634abf837c6661d977499e P 5e07d066bb864504b44c19f75dc565b7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 16:32:56.995064 2015 raft_consensus.cc:348] T 72d5dd9016634abf837c6661d977499e P 02ab510d1c094169b31adb687a154454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e07d066bb864504b44c19f75dc565b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44121 } } peers { permanent_uuid: "f1e3c666b63f43c796658fd44d19244e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46439 } } peers { permanent_uuid: "02ab510d1c094169b31adb687a154454" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42469 } }
I20260925 16:32:56.995168 2015 raft_consensus.cc:374] T 72d5dd9016634abf837c6661d977499e P 02ab510d1c094169b31adb687a154454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:32:56.995191 2015 raft_consensus.cc:729] T 72d5dd9016634abf837c6661d977499e P 02ab510d1c094169b31adb687a154454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02ab510d1c094169b31adb687a154454, State: Initialized, Role: FOLLOWER
I20260925 16:32:56.995285 2015 consensus_queue.cc:260] T 72d5dd9016634abf837c6661d977499e P 02ab510d1c094169b31adb687a154454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e07d066bb864504b44c19f75dc565b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44121 } } peers { permanent_uuid: "f1e3c666b63f43c796658fd44d19244e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46439 } } peers { permanent_uuid: "02ab510d1c094169b31adb687a154454" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42469 } }
I20260925 16:32:56.995289 2013 raft_consensus.cc:348] T 72d5dd9016634abf837c6661d977499e P 5e07d066bb864504b44c19f75dc565b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e07d066bb864504b44c19f75dc565b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44121 } } peers { permanent_uuid: "f1e3c666b63f43c796658fd44d19244e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46439 } } peers { permanent_uuid: "02ab510d1c094169b31adb687a154454" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42469 } }
I20260925 16:32:56.995396 2013 raft_consensus.cc:374] T 72d5dd9016634abf837c6661d977499e P 5e07d066bb864504b44c19f75dc565b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:32:56.995430 2013 raft_consensus.cc:729] T 72d5dd9016634abf837c6661d977499e P 5e07d066bb864504b44c19f75dc565b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e07d066bb864504b44c19f75dc565b7, State: Initialized, Role: FOLLOWER
I20260925 16:32:56.995508 2015 ts_tablet_manager.cc:1428] T 72d5dd9016634abf837c6661d977499e P 02ab510d1c094169b31adb687a154454: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 16:32:56.995482 2021 tablet_replica.cc:439] T 72d5dd9016634abf837c6661d977499e P 02ab510d1c094169b31adb687a154454: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e07d066bb864504b44c19f75dc565b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44121 } } peers { permanent_uuid: "f1e3c666b63f43c796658fd44d19244e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46439 } } peers { permanent_uuid: "02ab510d1c094169b31adb687a154454" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42469 } } }
I20260925 16:32:56.995546 2013 consensus_queue.cc:260] T 72d5dd9016634abf837c6661d977499e P 5e07d066bb864504b44c19f75dc565b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e07d066bb864504b44c19f75dc565b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44121 } } peers { permanent_uuid: "f1e3c666b63f43c796658fd44d19244e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46439 } } peers { permanent_uuid: "02ab510d1c094169b31adb687a154454" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42469 } }
I20260925 16:32:56.995609 2021 tablet_replica.cc:442] T 72d5dd9016634abf837c6661d977499e P 02ab510d1c094169b31adb687a154454: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 16:32:56.995790 2013 ts_tablet_manager.cc:1428] T 72d5dd9016634abf837c6661d977499e P 5e07d066bb864504b44c19f75dc565b7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 16:32:56.996017 2014 tablet_bootstrap.cc:492] T 72d5dd9016634abf837c6661d977499e P f1e3c666b63f43c796658fd44d19244e: No bootstrap required, opened a new log
I20260925 16:32:56.996094 2014 ts_tablet_manager.cc:1397] T 72d5dd9016634abf837c6661d977499e P f1e3c666b63f43c796658fd44d19244e: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260925 16:32:56.996086 2019 tablet_replica.cc:439] T 72d5dd9016634abf837c6661d977499e P 5e07d066bb864504b44c19f75dc565b7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e07d066bb864504b44c19f75dc565b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44121 } } peers { permanent_uuid: "f1e3c666b63f43c796658fd44d19244e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46439 } } peers { permanent_uuid: "02ab510d1c094169b31adb687a154454" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42469 } } }
I20260925 16:32:56.996196 2019 tablet_replica.cc:442] T 72d5dd9016634abf837c6661d977499e P 5e07d066bb864504b44c19f75dc565b7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 16:32:56.996587 2014 raft_consensus.cc:348] T 72d5dd9016634abf837c6661d977499e P f1e3c666b63f43c796658fd44d19244e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e07d066bb864504b44c19f75dc565b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44121 } } peers { permanent_uuid: "f1e3c666b63f43c796658fd44d19244e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46439 } } peers { permanent_uuid: "02ab510d1c094169b31adb687a154454" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42469 } }
I20260925 16:32:56.996701 2014 raft_consensus.cc:374] T 72d5dd9016634abf837c6661d977499e P f1e3c666b63f43c796658fd44d19244e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:32:56.996733 2014 raft_consensus.cc:729] T 72d5dd9016634abf837c6661d977499e P f1e3c666b63f43c796658fd44d19244e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e3c666b63f43c796658fd44d19244e, State: Initialized, Role: FOLLOWER
I20260925 16:32:56.996882 2014 consensus_queue.cc:260] T 72d5dd9016634abf837c6661d977499e P f1e3c666b63f43c796658fd44d19244e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e07d066bb864504b44c19f75dc565b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44121 } } peers { permanent_uuid: "f1e3c666b63f43c796658fd44d19244e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46439 } } peers { permanent_uuid: "02ab510d1c094169b31adb687a154454" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42469 } }
I20260925 16:32:56.997143 2020 tablet_replica.cc:439] T 72d5dd9016634abf837c6661d977499e P f1e3c666b63f43c796658fd44d19244e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e07d066bb864504b44c19f75dc565b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44121 } } peers { permanent_uuid: "f1e3c666b63f43c796658fd44d19244e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46439 } } peers { permanent_uuid: "02ab510d1c094169b31adb687a154454" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42469 } } }
I20260925 16:32:56.997279 2020 tablet_replica.cc:442] T 72d5dd9016634abf837c6661d977499e P f1e3c666b63f43c796658fd44d19244e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 16:32:56.997148 2014 ts_tablet_manager.cc:1428] T 72d5dd9016634abf837c6661d977499e P f1e3c666b63f43c796658fd44d19244e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 16:32:57.212159 2021 raft_consensus.cc:482] T 72d5dd9016634abf837c6661d977499e P 02ab510d1c094169b31adb687a154454 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:32:57.212324 2021 raft_consensus.cc:504] T 72d5dd9016634abf837c6661d977499e P 02ab510d1c094169b31adb687a154454 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e07d066bb864504b44c19f75dc565b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44121 } } peers { permanent_uuid: "f1e3c666b63f43c796658fd44d19244e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46439 } } peers { permanent_uuid: "02ab510d1c094169b31adb687a154454" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42469 } }
I20260925 16:32:57.212839 2021 leader_election.cc:288] T 72d5dd9016634abf837c6661d977499e P 02ab510d1c094169b31adb687a154454 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e07d066bb864504b44c19f75dc565b7 (127.0.98.1:44121), f1e3c666b63f43c796658fd44d19244e (127.0.98.2:46439)
I20260925 16:32:57.213349 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72d5dd9016634abf837c6661d977499e" candidate_uuid: "02ab510d1c094169b31adb687a154454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e07d066bb864504b44c19f75dc565b7" is_pre_election: true
I20260925 16:32:57.213344 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72d5dd9016634abf837c6661d977499e" candidate_uuid: "02ab510d1c094169b31adb687a154454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e3c666b63f43c796658fd44d19244e" is_pre_election: true
I20260925 16:32:57.213481 1664 raft_consensus.cc:2454] T 72d5dd9016634abf837c6661d977499e P 5e07d066bb864504b44c19f75dc565b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02ab510d1c094169b31adb687a154454 in term 0.
I20260925 16:32:57.213521 1804 raft_consensus.cc:2454] T 72d5dd9016634abf837c6661d977499e P f1e3c666b63f43c796658fd44d19244e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02ab510d1c094169b31adb687a154454 in term 0.
I20260925 16:32:57.213804 1877 leader_election.cc:302] T 72d5dd9016634abf837c6661d977499e P 02ab510d1c094169b31adb687a154454 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02ab510d1c094169b31adb687a154454, 5e07d066bb864504b44c19f75dc565b7; no voters:
I20260925 16:32:57.213989 2021 raft_consensus.cc:2781] T 72d5dd9016634abf837c6661d977499e P 02ab510d1c094169b31adb687a154454 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:32:57.214052 2021 raft_consensus.cc:482] T 72d5dd9016634abf837c6661d977499e P 02ab510d1c094169b31adb687a154454 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:32:57.214130 2021 raft_consensus.cc:3037] T 72d5dd9016634abf837c6661d977499e P 02ab510d1c094169b31adb687a154454 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:32:57.215130 2021 raft_consensus.cc:504] T 72d5dd9016634abf837c6661d977499e P 02ab510d1c094169b31adb687a154454 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e07d066bb864504b44c19f75dc565b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44121 } } peers { permanent_uuid: "f1e3c666b63f43c796658fd44d19244e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46439 } } peers { permanent_uuid: "02ab510d1c094169b31adb687a154454" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42469 } }
I20260925 16:32:57.215513 2021 leader_election.cc:288] T 72d5dd9016634abf837c6661d977499e P 02ab510d1c094169b31adb687a154454 [CANDIDATE]: Term 1 election: Requested vote from peers 5e07d066bb864504b44c19f75dc565b7 (127.0.98.1:44121), f1e3c666b63f43c796658fd44d19244e (127.0.98.2:46439)
I20260925 16:32:57.215857 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72d5dd9016634abf837c6661d977499e" candidate_uuid: "02ab510d1c094169b31adb687a154454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e07d066bb864504b44c19f75dc565b7"
I20260925 16:32:57.215883 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72d5dd9016634abf837c6661d977499e" candidate_uuid: "02ab510d1c094169b31adb687a154454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e3c666b63f43c796658fd44d19244e"
I20260925 16:32:57.215993 1804 raft_consensus.cc:3037] T 72d5dd9016634abf837c6661d977499e P f1e3c666b63f43c796658fd44d19244e [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:32:57.215986 1664 raft_consensus.cc:3037] T 72d5dd9016634abf837c6661d977499e P 5e07d066bb864504b44c19f75dc565b7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:32:57.216837 1804 raft_consensus.cc:2454] T 72d5dd9016634abf837c6661d977499e P f1e3c666b63f43c796658fd44d19244e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02ab510d1c094169b31adb687a154454 in term 1.
I20260925 16:32:57.216975 1664 raft_consensus.cc:2454] T 72d5dd9016634abf837c6661d977499e P 5e07d066bb864504b44c19f75dc565b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02ab510d1c094169b31adb687a154454 in term 1.
I20260925 16:32:57.217139 1875 leader_election.cc:302] T 72d5dd9016634abf837c6661d977499e P 02ab510d1c094169b31adb687a154454 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02ab510d1c094169b31adb687a154454, f1e3c666b63f43c796658fd44d19244e; no voters:
I20260925 16:32:57.217320 2021 raft_consensus.cc:2781] T 72d5dd9016634abf837c6661d977499e P 02ab510d1c094169b31adb687a154454 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:32:57.217512 2021 raft_consensus.cc:686] T 72d5dd9016634abf837c6661d977499e P 02ab510d1c094169b31adb687a154454 [term 1 LEADER]: Becoming Leader. State: Replica: 02ab510d1c094169b31adb687a154454, State: Running, Role: LEADER
I20260925 16:32:57.217658 2021 consensus_queue.cc:237] T 72d5dd9016634abf837c6661d977499e P 02ab510d1c094169b31adb687a154454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e07d066bb864504b44c19f75dc565b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44121 } } peers { permanent_uuid: "f1e3c666b63f43c796658fd44d19244e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46439 } } peers { permanent_uuid: "02ab510d1c094169b31adb687a154454" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42469 } }
I20260925 16:32:57.218343 2055 tablet_replica.cc:439] T 72d5dd9016634abf837c6661d977499e P 02ab510d1c094169b31adb687a154454: TxnStatusTablet state changed. Reason: New leader 02ab510d1c094169b31adb687a154454. Latest consensus state: current_term: 1 leader_uuid: "02ab510d1c094169b31adb687a154454" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e07d066bb864504b44c19f75dc565b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44121 } } peers { permanent_uuid: "f1e3c666b63f43c796658fd44d19244e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46439 } } peers { permanent_uuid: "02ab510d1c094169b31adb687a154454" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42469 } } }
I20260925 16:32:57.218475 2055 tablet_replica.cc:442] T 72d5dd9016634abf837c6661d977499e P 02ab510d1c094169b31adb687a154454: This TxnStatusTablet replica's current role is: LEADER
I20260925 16:32:57.218698 2057 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 16:32:57.218878 2057 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 16:32:57.219511 1506 catalog_manager.cc:5495] T 72d5dd9016634abf837c6661d977499e P 02ab510d1c094169b31adb687a154454 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02ab510d1c094169b31adb687a154454 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "02ab510d1c094169b31adb687a154454" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e07d066bb864504b44c19f75dc565b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1e3c666b63f43c796658fd44d19244e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02ab510d1c094169b31adb687a154454" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42469 } health_report { overall_health: HEALTHY } } }
I20260925 16:32:57.233445 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:57.229834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55094 (local address 127.0.98.3:42469)
0925 16:32:57.229956 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:57.229959 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:57.230049 (+ 90us) server_negotiation.cc:407] Connection header received
0925 16:32:57.230208 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:57.230242 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:57.230290 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:57.230380 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:32:57.230666 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:57.231166 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:57.232359 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:57.232568 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:57.232646 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:57.232960 (+ 314us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:32:57.233031 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:32:57.233087 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:57.233287 (+ 200us) server_negotiation.cc:299] Negotiation successful
0925 16:32:57.233341 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 16:32:57.233443 2050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:57.229740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42469 (local address 127.0.0.1:55094)
0925 16:32:57.229973 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:57.229988 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:32:57.230096 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:32:57.230423 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:32:57.230426 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:32:57.230433 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 16:32:57.230544 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:57.230549 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:57.231267 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:57.231270 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:57.232237 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:57.232247 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:57.232670 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:57.232674 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:57.232770 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:57.232825 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 16:32:57.233112 (+ 287us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 16:32:57.233126 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 16:32:57.233192 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 16:32:57.233279 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135}
I20260925 16:32:57.236136 1804 raft_consensus.cc:1261] T 72d5dd9016634abf837c6661d977499e P f1e3c666b63f43c796658fd44d19244e [term 1 FOLLOWER]: Refusing update from remote peer 02ab510d1c094169b31adb687a154454: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:32:57.236191 1664 raft_consensus.cc:1261] T 72d5dd9016634abf837c6661d977499e P 5e07d066bb864504b44c19f75dc565b7 [term 1 FOLLOWER]: Refusing update from remote peer 02ab510d1c094169b31adb687a154454: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:32:57.236580 2021 consensus_queue.cc:1035] T 72d5dd9016634abf837c6661d977499e P 02ab510d1c094169b31adb687a154454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e3c666b63f43c796658fd44d19244e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:32:57.236775 2055 consensus_queue.cc:1035] T 72d5dd9016634abf837c6661d977499e P 02ab510d1c094169b31adb687a154454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e07d066bb864504b44c19f75dc565b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:32:57.238047 2020 tablet_replica.cc:439] T 72d5dd9016634abf837c6661d977499e P f1e3c666b63f43c796658fd44d19244e: TxnStatusTablet state changed. Reason: New leader 02ab510d1c094169b31adb687a154454. Latest consensus state: current_term: 1 leader_uuid: "02ab510d1c094169b31adb687a154454" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e07d066bb864504b44c19f75dc565b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44121 } } peers { permanent_uuid: "f1e3c666b63f43c796658fd44d19244e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46439 } } peers { permanent_uuid: "02ab510d1c094169b31adb687a154454" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42469 } } }
I20260925 16:32:57.238186 2020 tablet_replica.cc:442] T 72d5dd9016634abf837c6661d977499e P f1e3c666b63f43c796658fd44d19244e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 16:32:57.238373 2019 tablet_replica.cc:439] T 72d5dd9016634abf837c6661d977499e P 5e07d066bb864504b44c19f75dc565b7: TxnStatusTablet state changed. Reason: New leader 02ab510d1c094169b31adb687a154454. Latest consensus state: current_term: 1 leader_uuid: "02ab510d1c094169b31adb687a154454" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e07d066bb864504b44c19f75dc565b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44121 } } peers { permanent_uuid: "f1e3c666b63f43c796658fd44d19244e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46439 } } peers { permanent_uuid: "02ab510d1c094169b31adb687a154454" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42469 } } }
I20260925 16:32:57.238485 2019 tablet_replica.cc:442] T 72d5dd9016634abf837c6661d977499e P 5e07d066bb864504b44c19f75dc565b7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 16:32:57.241173 2055 tablet_replica.cc:439] T 72d5dd9016634abf837c6661d977499e P 02ab510d1c094169b31adb687a154454: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "02ab510d1c094169b31adb687a154454" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e07d066bb864504b44c19f75dc565b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44121 } } peers { permanent_uuid: "f1e3c666b63f43c796658fd44d19244e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46439 } } peers { permanent_uuid: "02ab510d1c094169b31adb687a154454" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42469 } } }
I20260925 16:32:57.241374 2055 tablet_replica.cc:442] T 72d5dd9016634abf837c6661d977499e P 02ab510d1c094169b31adb687a154454: This TxnStatusTablet replica's current role is: LEADER
I20260925 16:32:57.241541 2055 tablet_replica.cc:439] T 72d5dd9016634abf837c6661d977499e P 02ab510d1c094169b31adb687a154454: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "02ab510d1c094169b31adb687a154454" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e07d066bb864504b44c19f75dc565b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44121 } } peers { permanent_uuid: "f1e3c666b63f43c796658fd44d19244e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46439 } } peers { permanent_uuid: "02ab510d1c094169b31adb687a154454" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42469 } } }
I20260925 16:32:57.241671 2055 tablet_replica.cc:442] T 72d5dd9016634abf837c6661d977499e P 02ab510d1c094169b31adb687a154454: This TxnStatusTablet replica's current role is: LEADER
I20260925 16:32:57.241987 2019 tablet_replica.cc:439] T 72d5dd9016634abf837c6661d977499e P 5e07d066bb864504b44c19f75dc565b7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "02ab510d1c094169b31adb687a154454" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e07d066bb864504b44c19f75dc565b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44121 } } peers { permanent_uuid: "f1e3c666b63f43c796658fd44d19244e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46439 } } peers { permanent_uuid: "02ab510d1c094169b31adb687a154454" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42469 } } }
I20260925 16:32:57.242123 2019 tablet_replica.cc:442] T 72d5dd9016634abf837c6661d977499e P 5e07d066bb864504b44c19f75dc565b7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 16:32:57.242061 2020 tablet_replica.cc:439] T 72d5dd9016634abf837c6661d977499e P f1e3c666b63f43c796658fd44d19244e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "02ab510d1c094169b31adb687a154454" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e07d066bb864504b44c19f75dc565b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44121 } } peers { permanent_uuid: "f1e3c666b63f43c796658fd44d19244e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46439 } } peers { permanent_uuid: "02ab510d1c094169b31adb687a154454" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42469 } } }
I20260925 16:32:57.242164 2020 tablet_replica.cc:442] T 72d5dd9016634abf837c6661d977499e P f1e3c666b63f43c796658fd44d19244e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 16:32:57.244282 2060 mvcc.cc:205] Tried to move back new op lower bound from 7333289890755178496 to 7333289890685112320. Current Snapshot: MvccSnapshot[applied={T|T < 7333289890755178496 or (T in {7333289890755178496})}]
I20260925 16:32:57.252170 2071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:57.247975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47818 (local address 127.0.98.1:44121)
0925 16:32:57.248171 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:57.248175 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:57.248190 (+ 15us) server_negotiation.cc:407] Connection header received
0925 16:32:57.248331 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:57.248335 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:57.248406 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:57.248533 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:32:57.248736 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:57.249545 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:57.250399 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:57.250782 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:57.250887 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:57.251169 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:32:57.251758 (+ 589us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:32:57.251871 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:57.251944 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 16:32:57.252004 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":60,"threads_started":1}
I20260925 16:32:57.263453 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:57.259573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52499 (local address 127.0.98.3:42469)
0925 16:32:57.259712 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:57.259716 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:57.259854 (+ 138us) server_negotiation.cc:407] Connection header received
0925 16:32:57.260001 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:57.260004 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:57.260061 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:57.260130 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:32:57.260365 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:57.261089 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:57.262412 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:57.262662 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:57.262755 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:57.263045 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:32:57.263112 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:32:57.263167 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:57.263290 (+ 123us) server_negotiation.cc:299] Negotiation successful
0925 16:32:57.263339 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 16:32:57.263513 2074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:57.259427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42469 (local address 127.0.98.1:52499)
0925 16:32:57.259781 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:57.259796 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:32:57.259896 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:32:57.260157 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:32:57.260159 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:32:57.260167 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 16:32:57.260265 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:57.260270 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:57.261214 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:57.261218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:57.262244 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:57.262255 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:57.262771 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:57.262774 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:57.262869 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:57.262918 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 16:32:57.263198 (+ 280us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 16:32:57.263204 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 16:32:57.263271 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 16:32:57.263345 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":116,"threads_started":1}
I20260925 16:32:57.271052 1804 raft_consensus.cc:1261] T 9c984fdf71cc471386fffea9daeab895 P f1e3c666b63f43c796658fd44d19244e [term 1 FOLLOWER]: Refusing update from remote peer 5e07d066bb864504b44c19f75dc565b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:32:57.271045 1944 raft_consensus.cc:1261] T 9c984fdf71cc471386fffea9daeab895 P 02ab510d1c094169b31adb687a154454 [term 1 FOLLOWER]: Refusing update from remote peer 5e07d066bb864504b44c19f75dc565b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:32:57.271534 2043 consensus_queue.cc:1035] T 9c984fdf71cc471386fffea9daeab895 P 5e07d066bb864504b44c19f75dc565b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e3c666b63f43c796658fd44d19244e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:32:57.271714 2019 consensus_queue.cc:1035] T 9c984fdf71cc471386fffea9daeab895 P 5e07d066bb864504b44c19f75dc565b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02ab510d1c094169b31adb687a154454" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:32:57.295603 2078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:57.291458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44121 (local address 127.0.98.3:57339)
0925 16:32:57.291857 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:57.291870 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:32:57.291957 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:32:57.292199 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:32:57.292202 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:32:57.292209 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 16:32:57.292351 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:57.292357 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:57.293378 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:57.293381 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:57.294348 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:57.294358 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:57.294950 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:57.294953 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:57.295018 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:57.295060 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 16:32:57.295352 (+ 292us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 16:32:57.295358 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 16:32:57.295412 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 16:32:57.295477 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":109,"threads_started":1}
I20260925 16:32:57.295603 2071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:57.291576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57339 (local address 127.0.98.1:44121)
0925 16:32:57.291705 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:57.291710 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:57.291939 (+ 229us) server_negotiation.cc:407] Connection header received
0925 16:32:57.292030 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:57.292033 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:57.292082 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:57.292162 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:32:57.292489 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:57.293268 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:57.294463 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:57.294824 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:57.294934 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:57.295186 (+ 252us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:32:57.295266 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:32:57.295337 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:57.295424 (+ 87us) server_negotiation.cc:299] Negotiation successful
0925 16:32:57.295476 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 16:32:57.324213 1506 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:32:57.324407 1506 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:32:57.326719 1506 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eec9855ba8004aecae4b7c47ecce51d3: Sending DeleteTablet for 3 replicas of tablet 9c984fdf71cc471386fffea9daeab895
I20260925 16:32:57.327351 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c984fdf71cc471386fffea9daeab895 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:32:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55070
I20260925 16:32:57.327687 1784 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c984fdf71cc471386fffea9daeab895 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:32:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56058
I20260925 16:32:57.328069 2083 tablet_replica.cc:330] T 9c984fdf71cc471386fffea9daeab895 P f1e3c666b63f43c796658fd44d19244e: stopping tablet replica
I20260925 16:32:57.328202 2083 raft_consensus.cc:2229] T 9c984fdf71cc471386fffea9daeab895 P f1e3c666b63f43c796658fd44d19244e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:32:57.328392 2083 raft_consensus.cc:2258] T 9c984fdf71cc471386fffea9daeab895 P f1e3c666b63f43c796658fd44d19244e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:32:57.329097 2083 ts_tablet_manager.cc:1905] T 9c984fdf71cc471386fffea9daeab895 P f1e3c666b63f43c796658fd44d19244e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:32:57.329257 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c984fdf71cc471386fffea9daeab895 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:32:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47804
I20260925 16:32:57.330099 2085 tablet_replica.cc:330] T 9c984fdf71cc471386fffea9daeab895 P 5e07d066bb864504b44c19f75dc565b7: stopping tablet replica
I20260925 16:32:57.330261 2085 raft_consensus.cc:2229] T 9c984fdf71cc471386fffea9daeab895 P 5e07d066bb864504b44c19f75dc565b7 [term 1 LEADER]: Raft consensus shutting down.
I20260925 16:32:57.330904 2085 raft_consensus.cc:2258] T 9c984fdf71cc471386fffea9daeab895 P 5e07d066bb864504b44c19f75dc565b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:32:57.331590 2085 ts_tablet_manager.cc:1905] T 9c984fdf71cc471386fffea9daeab895 P 5e07d066bb864504b44c19f75dc565b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:32:57.332048 2083 ts_tablet_manager.cc:1918] T 9c984fdf71cc471386fffea9daeab895 P f1e3c666b63f43c796658fd44d19244e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 16:32:57.332129 2083 log.cc:1196] T 9c984fdf71cc471386fffea9daeab895 P f1e3c666b63f43c796658fd44d19244e: Deleting WAL directory at /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/minicluster-data/ts-1/wal/wals/9c984fdf71cc471386fffea9daeab895
I20260925 16:32:57.332326 2082 tablet_replica.cc:330] T 9c984fdf71cc471386fffea9daeab895 P 02ab510d1c094169b31adb687a154454: stopping tablet replica
I20260925 16:32:57.332397 2083 ts_tablet_manager.cc:1939] T 9c984fdf71cc471386fffea9daeab895 P f1e3c666b63f43c796658fd44d19244e: Deleting consensus metadata
I20260925 16:32:57.333052 1491 catalog_manager.cc:4841] TS f1e3c666b63f43c796658fd44d19244e (127.0.98.2:46439): tablet 9c984fdf71cc471386fffea9daeab895 (table test-table [id=72b11d68f4f54d96814aa4c2faf10ea5]) successfully deleted
I20260925 16:32:57.333096 2082 raft_consensus.cc:2229] T 9c984fdf71cc471386fffea9daeab895 P 02ab510d1c094169b31adb687a154454 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:32:57.333277 2082 raft_consensus.cc:2258] T 9c984fdf71cc471386fffea9daeab895 P 02ab510d1c094169b31adb687a154454 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:32:57.333841 2082 ts_tablet_manager.cc:1905] T 9c984fdf71cc471386fffea9daeab895 P 02ab510d1c094169b31adb687a154454: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:32:57.334625 2085 ts_tablet_manager.cc:1918] T 9c984fdf71cc471386fffea9daeab895 P 5e07d066bb864504b44c19f75dc565b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 16:32:57.334719 2085 log.cc:1196] T 9c984fdf71cc471386fffea9daeab895 P 5e07d066bb864504b44c19f75dc565b7: Deleting WAL directory at /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/minicluster-data/ts-0/wal/wals/9c984fdf71cc471386fffea9daeab895
I20260925 16:32:57.334982 2085 ts_tablet_manager.cc:1939] T 9c984fdf71cc471386fffea9daeab895 P 5e07d066bb864504b44c19f75dc565b7: Deleting consensus metadata
I20260925 16:32:57.335592 1493 catalog_manager.cc:4841] TS 5e07d066bb864504b44c19f75dc565b7 (127.0.98.1:44121): tablet 9c984fdf71cc471386fffea9daeab895 (table test-table [id=72b11d68f4f54d96814aa4c2faf10ea5]) successfully deleted
I20260925 16:32:57.336375 2082 ts_tablet_manager.cc:1918] T 9c984fdf71cc471386fffea9daeab895 P 02ab510d1c094169b31adb687a154454: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 16:32:57.336449 2082 log.cc:1196] T 9c984fdf71cc471386fffea9daeab895 P 02ab510d1c094169b31adb687a154454: Deleting WAL directory at /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353966814071-392-0/minicluster-data/ts-2/wal/wals/9c984fdf71cc471386fffea9daeab895
I20260925 16:32:57.336711 2082 ts_tablet_manager.cc:1939] T 9c984fdf71cc471386fffea9daeab895 P 02ab510d1c094169b31adb687a154454: Deleting consensus metadata
I20260925 16:32:57.337422 1493 catalog_manager.cc:4841] TS 02ab510d1c094169b31adb687a154454 (127.0.98.3:42469): tablet 9c984fdf71cc471386fffea9daeab895 (table test-table [id=72b11d68f4f54d96814aa4c2faf10ea5]) successfully deleted
Sep 25 16:32:57 dist-test-slave-65pf krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353976, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 16:32:57.346537 2071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:57.329848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47830 (local address 127.0.98.1:44121)
0925 16:32:57.329957 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:57.329961 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:57.331282 (+ 1321us) server_negotiation.cc:407] Connection header received
0925 16:32:57.331468 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:57.331471 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:57.331513 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:57.331567 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:57.333465 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:57.334213 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:57.337974 (+ 3761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:57.338349 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:57.338463 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:57.342407 (+ 3944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:32:57.342417 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:32:57.342432 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:32:57.342459 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:32:57.344634 (+ 2175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:57.345123 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:57.345130 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:57.345132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:57.345216 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:57.345507 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:57.345510 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:57.345512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:57.345807 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:32:57.346004 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:57.346236 (+ 232us) server_negotiation.cc:299] Negotiation successful
0925 16:32:57.346360 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260925 16:32:57.347031 1704 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47830
I20260925 16:32:57.360740 2051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:57.351024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52972 (local address 127.0.98.62:45269)
0925 16:32:57.351165 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:57.351169 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:57.351319 (+ 150us) server_negotiation.cc:407] Connection header received
0925 16:32:57.351459 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:57.351462 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:57.351518 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:57.351612 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:57.352456 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:57.353297 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:57.354427 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:57.354857 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:57.354939 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:57.356769 (+ 1830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:32:57.356773 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:32:57.356776 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:32:57.356804 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:32:57.358569 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:57.359097 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:57.359101 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:57.359103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:57.359203 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:57.359625 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:57.359629 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:57.359630 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:57.359873 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:32:57.359991 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:57.360418 (+ 427us) server_negotiation.cc:299] Negotiation successful
0925 16:32:57.360554 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260925 16:32:57.362425 1506 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52972
I20260925 16:32:57.364514 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 1579
I20260925 16:32:57.373606 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 1719
I20260925 16:32:57.380455 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 1859
I20260925 16:32:57.385923 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 1475
2026-09-25T16:32:57Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3110 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:32:57 dist-test-slave-65pf krb5kdc[2091](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:32:57 dist-test-slave-65pf krb5kdc[2091](info): set up 2 sockets
Sep 25 16:32:57 dist-test-slave-65pf krb5kdc[2091](info): commencing operation
krb5kdc: starting...
W20260925 16:32:58.438464 392 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/krb5kdc/test-user.keytab.
Sep 25 16:32:58 dist-test-slave-65pf krb5kdc[2091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:32:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:32:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 16:32:58.565335 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45895
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45895
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:32:58.673884 2104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:32:58.674196 2104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:32:58.674275 2104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:32:58.678347 2104 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:32:58.678442 2104 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 16:32:58.678474 2104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:32:58.678517 2104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:32:58.678557 2104 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:32:58.678628 2104 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:32:58.683985 2104 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45895
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:32:58.685057 2104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:32:58.686062 2104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:32:58.694410 2110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:32:58.694439 2109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:32:58.694638 2104 server_base.cc:1034] running on GCE node
W20260925 16:32:58.694451 2112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:32:58.695353 2104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:32:58.696483 2104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:32:58.697669 2104 hybrid_clock.cc:648] HybridClock initialized: now 1790353978697636 us; error 48 us; skew 500 ppm
Sep 25 16:32:58 dist-test-slave-65pf krb5kdc[2091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353978, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:32:58.700430 2104 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 16:32:58.701393 2104 webserver.cc:460] Webserver started at http://127.0.98.62:46531/ using document root <none> and password file <none>
I20260925 16:32:58.701957 2104 fs_manager.cc:359] Metadata directory not provided
I20260925 16:32:58.702049 2104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:32:58.702280 2104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:32:58.704403 2104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/master-0/data/instance:
uuid: "2325bb98933b48e880649d64cbdac636"
format_stamp: "Formatted at 2026-09-25 16:32:58 on dist-test-slave-65pf"
server_key: "f701f6c56ef991f13dc5b07e47a7f42e"
server_key_iv: "0cc88cb2967c910471842728e1dfd11d"
server_key_version: "encryptionkey@0"
I20260925 16:32:58.704983 2104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/master-0/wal/instance:
uuid: "2325bb98933b48e880649d64cbdac636"
format_stamp: "Formatted at 2026-09-25 16:32:58 on dist-test-slave-65pf"
server_key: "f701f6c56ef991f13dc5b07e47a7f42e"
server_key_iv: "0cc88cb2967c910471842728e1dfd11d"
server_key_version: "encryptionkey@0"
I20260925 16:32:58.708779 2104 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 16:32:58.711221 2119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:32:58.712421 2104 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:32:58.712563 2104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/master-0/wal
uuid: "2325bb98933b48e880649d64cbdac636"
format_stamp: "Formatted at 2026-09-25 16:32:58 on dist-test-slave-65pf"
server_key: "f701f6c56ef991f13dc5b07e47a7f42e"
server_key_iv: "0cc88cb2967c910471842728e1dfd11d"
server_key_version: "encryptionkey@0"
I20260925 16:32:58.712672 2104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:32:58.734169 2104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:32:58.737589 2104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:32:58.737802 2104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:32:58.746470 2104 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45895
I20260925 16:32:58.746477 2181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45895 every 8 connection(s)
I20260925 16:32:58.747449 2104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:32:58.750145 2182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:32:58.751791 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 2104
I20260925 16:32:58.751902 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/master-0/wal/instance
I20260925 16:32:58.752167 392 external_mini_cluster.cc:1428] Setting key dd2bdcef44d3bbdb17ef9a546d8dde04
I20260925 16:32:58.756858 2182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2325bb98933b48e880649d64cbdac636: Bootstrap starting.
I20260925 16:32:58.759076 2182 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2325bb98933b48e880649d64cbdac636: Neither blocks nor log segments found. Creating new log.
I20260925 16:32:58.760021 2182 log.cc:824] T 00000000000000000000000000000000 P 2325bb98933b48e880649d64cbdac636: Log is configured to *not* fsync() on all Append() calls
Sep 25 16:32:58 dist-test-slave-65pf krb5kdc[2091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:32:58.762230 2182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2325bb98933b48e880649d64cbdac636: No bootstrap required, opened a new log
I20260925 16:32:58.766304 2182 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2325bb98933b48e880649d64cbdac636 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2325bb98933b48e880649d64cbdac636" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45895 } }
I20260925 16:32:58.766569 2182 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2325bb98933b48e880649d64cbdac636 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:32:58.766651 2182 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2325bb98933b48e880649d64cbdac636 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2325bb98933b48e880649d64cbdac636, State: Initialized, Role: FOLLOWER
I20260925 16:32:58.767102 2182 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2325bb98933b48e880649d64cbdac636 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2325bb98933b48e880649d64cbdac636" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45895 } }
I20260925 16:32:58.767192 2185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:58.753442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55662 (local address 127.0.98.62:45895)
0925 16:32:58.753844 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:58.753858 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:58.753895 (+ 37us) server_negotiation.cc:407] Connection header received
0925 16:32:58.754522 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:58.754538 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:58.754967 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:58.755357 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:58.756140 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:58.757160 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:58.757997 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:58.758471 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:58.758676 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:58.761579 (+ 2903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:32:58.761585 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:32:58.761601 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:32:58.761631 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:32:58.764208 (+ 2577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:58.764615 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:58.764621 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:58.764626 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:58.764761 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:58.765010 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:58.765014 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:58.765016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:58.765506 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:32:58.765725 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:58.765902 (+ 177us) server_negotiation.cc:299] Negotiation successful
0925 16:32:58.766182 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":105,"threads_started":1}
I20260925 16:32:58.767282 2182 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2325bb98933b48e880649d64cbdac636 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:32:58.767338 2182 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2325bb98933b48e880649d64cbdac636 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:32:58.767432 2182 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2325bb98933b48e880649d64cbdac636 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:32:58.768500 2182 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2325bb98933b48e880649d64cbdac636 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2325bb98933b48e880649d64cbdac636" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45895 } }
I20260925 16:32:58.768831 2182 leader_election.cc:302] T 00000000000000000000000000000000 P 2325bb98933b48e880649d64cbdac636 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2325bb98933b48e880649d64cbdac636; no voters:
I20260925 16:32:58.769120 2182 leader_election.cc:288] T 00000000000000000000000000000000 P 2325bb98933b48e880649d64cbdac636 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:32:58.769224 2187 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2325bb98933b48e880649d64cbdac636 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:32:58.769445 2187 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2325bb98933b48e880649d64cbdac636 [term 1 LEADER]: Becoming Leader. State: Replica: 2325bb98933b48e880649d64cbdac636, State: Running, Role: LEADER
I20260925 16:32:58.769739 2187 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2325bb98933b48e880649d64cbdac636 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2325bb98933b48e880649d64cbdac636" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45895 } }
I20260925 16:32:58.770339 2182 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2325bb98933b48e880649d64cbdac636 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:32:58.771459 2189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2325bb98933b48e880649d64cbdac636 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2325bb98933b48e880649d64cbdac636. Latest consensus state: current_term: 1 leader_uuid: "2325bb98933b48e880649d64cbdac636" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2325bb98933b48e880649d64cbdac636" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45895 } } }
I20260925 16:32:58.771471 2188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2325bb98933b48e880649d64cbdac636 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2325bb98933b48e880649d64cbdac636" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2325bb98933b48e880649d64cbdac636" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45895 } } }
I20260925 16:32:58.771608 2188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2325bb98933b48e880649d64cbdac636 [sys.catalog]: This master's current role is: LEADER
I20260925 16:32:58.771606 2189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2325bb98933b48e880649d64cbdac636 [sys.catalog]: This master's current role is: LEADER
I20260925 16:32:58.772693 2196 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:32:58.775664 2196 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:32:58.781333 2196 catalog_manager.cc:1347] Generated new cluster ID: 994030f07b694e9f8216e759488765e8
I20260925 16:32:58.781417 2196 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:32:58.789002 2196 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:32:58.789913 2196 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:32:58.797698 2196 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2325bb98933b48e880649d64cbdac636: Generated new TSK 0
I20260925 16:32:58.798385 2196 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 16:32:58.828876 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45895
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:32:58.940876 2208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:32:58.941124 2208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:32:58.941169 2208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:32:58.945724 2208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:32:58.945778 2208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:32:58.945819 2208 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 16:32:58.945833 2208 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 16:32:58.945865 2208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 16:32:58.951097 2208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:32:58.952072 2208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:32:58.953037 2208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:32:58.962549 2216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:32:58.962544 2214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:32:58.962544 2213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:32:58.962615 2208 server_base.cc:1034] running on GCE node
I20260925 16:32:58.963196 2208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:32:58.963811 2208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:32:58.964993 2208 hybrid_clock.cc:648] HybridClock initialized: now 1790353978964958 us; error 51 us; skew 500 ppm
Sep 25 16:32:58 dist-test-slave-65pf krb5kdc[2091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353978, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:32:58.967773 2208 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 16:32:58.968609 2208 webserver.cc:460] Webserver started at http://127.0.98.1:36365/ using document root <none> and password file <none>
I20260925 16:32:58.969187 2208 fs_manager.cc:359] Metadata directory not provided
I20260925 16:32:58.969278 2208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:32:58.969501 2208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:32:58.971694 2208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/ts-0/data/instance:
uuid: "435b9fb0fe93430face8f32e4de081b4"
format_stamp: "Formatted at 2026-09-25 16:32:58 on dist-test-slave-65pf"
server_key: "1c635a8f0e34f109a63ffc06f1f486b2"
server_key_iv: "ed5d6f7fc14343d9263c9c53e962b62f"
server_key_version: "encryptionkey@0"
I20260925 16:32:58.972256 2208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/ts-0/wal/instance:
uuid: "435b9fb0fe93430face8f32e4de081b4"
format_stamp: "Formatted at 2026-09-25 16:32:58 on dist-test-slave-65pf"
server_key: "1c635a8f0e34f109a63ffc06f1f486b2"
server_key_iv: "ed5d6f7fc14343d9263c9c53e962b62f"
server_key_version: "encryptionkey@0"
I20260925 16:32:58.975785 2208 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 16:32:58.978035 2223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:32:58.979240 2208 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:32:58.979372 2208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/ts-0/wal
uuid: "435b9fb0fe93430face8f32e4de081b4"
format_stamp: "Formatted at 2026-09-25 16:32:58 on dist-test-slave-65pf"
server_key: "1c635a8f0e34f109a63ffc06f1f486b2"
server_key_iv: "ed5d6f7fc14343d9263c9c53e962b62f"
server_key_version: "encryptionkey@0"
I20260925 16:32:58.979472 2208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:32:58.996917 2208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:32:58.999766 2208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:32:58.999976 2208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:32:59.001495 2208 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:32:59.001566 2208 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:32:59.001642 2208 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:32:59.001695 2208 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 16:32:59 dist-test-slave-65pf krb5kdc[2091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353978, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:32:59.012563 2208 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43805
I20260925 16:32:59.012871 2338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43805 every 8 connection(s)
I20260925 16:32:59.013511 2208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
I20260925 16:32:59.015165 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 2208
I20260925 16:32:59.015278 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/ts-0/wal/instance
I20260925 16:32:59.015578 392 external_mini_cluster.cc:1428] Setting key 364970a5241edb238c15d62cdbdeac98
I20260925 16:32:59.016310 2185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:59.002277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45113 (local address 127.0.98.62:45895)
0925 16:32:59.002413 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:59.002417 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:59.003483 (+ 1066us) server_negotiation.cc:407] Connection header received
0925 16:32:59.004617 (+ 1134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:59.004621 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:59.004683 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:59.004741 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:59.006495 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:59.007447 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:59.008985 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:59.009364 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:59.009483 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:59.012252 (+ 2769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:32:59.012257 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:32:59.012260 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:32:59.012290 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:32:59.014069 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:59.014629 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:59.014633 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:59.014635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:59.014730 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:59.015053 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:59.015058 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:59.015061 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:59.015321 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:32:59.015421 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:59.016015 (+ 594us) server_negotiation.cc:299] Negotiation successful
0925 16:32:59.016133 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 16:32:59.017297 2232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:59.002881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45895 (local address 127.0.98.1:45113)
0925 16:32:59.003338 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:59.003381 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 16:32:59.004363 (+ 982us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:32:59.005183 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:32:59.005193 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:32:59.005610 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:32:59.006318 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:59.006340 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:59.007593 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:59.007597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:59.008855 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:59.008865 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:59.009520 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:59.009524 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:59.009711 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:59.010340 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:32:59.010378 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:32:59.012077 (+ 1699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:32:59.014203 (+ 2126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:59.014209 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:59.014224 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:59.014508 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:59.014818 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:59.014820 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:59.014821 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:59.014952 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:59.015447 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:32:59.015456 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:32:59.015783 (+ 327us) client_negotiation.cc:769] Sending connection context
0925 16:32:59.015943 (+ 160us) client_negotiation.cc:240] Negotiation successful
0925 16:32:59.016289 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":123,"threads_started":1}
I20260925 16:32:59.018723 2339 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45895
I20260925 16:32:59.018880 2339 heartbeater.cc:464] Registering TS with master...
I20260925 16:32:59.019454 2339 heartbeater.cc:511] Master 127.0.98.62:45895 requested a full tablet report, sending...
I20260925 16:32:59.021088 2135 ts_manager.cc:194] Registered new tserver with Master: 435b9fb0fe93430face8f32e4de081b4 (127.0.98.1:43805)
I20260925 16:32:59.022852 2135 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45113
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260925 16:32:59.028461 2346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:59.020793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45895 (local address 127.0.98.1:35605)
0925 16:32:59.021051 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:59.021066 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:32:59.021153 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:32:59.021493 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:32:59.021496 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:32:59.021689 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:32:59.021787 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:59.021792 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:59.022798 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:59.022801 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:59.023444 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:59.023473 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:59.024083 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:59.024087 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:59.024178 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:59.024745 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:32:59.024759 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:32:59.025181 (+ 422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:32:59.027004 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:59.027007 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:59.027008 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:59.027199 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:59.027435 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:59.027437 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:59.027438 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:59.027521 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:59.027954 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:32:59.027956 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:32:59.028032 (+ 76us) client_negotiation.cc:769] Sending connection context
0925 16:32:59.028117 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 16:32:59.028229 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":104,"threads_started":1}
I20260925 16:32:59.028548 2185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:59.021066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35605 (local address 127.0.98.62:45895)
0925 16:32:59.021180 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:59.021186 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:59.021201 (+ 15us) server_negotiation.cc:407] Connection header received
0925 16:32:59.021239 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:59.021242 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:59.021278 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:59.021375 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:59.021905 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:59.022673 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:59.023591 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:59.023955 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:59.024080 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:59.025330 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:32:59.025336 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:32:59.025343 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:32:59.025373 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:32:59.026907 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:59.027283 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:59.027286 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:59.027287 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:59.027356 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:59.027628 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:59.027633 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:59.027634 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:59.027869 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:32:59.027945 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:59.028205 (+ 260us) server_negotiation.cc:299] Negotiation successful
0925 16:32:59.028345 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":187,"server-negotiator.queue_time_us":48}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 16:32:59.045817 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45895
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:32:59.157184 2348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:32:59.157464 2348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:32:59.157513 2348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:32:59.161791 2348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:32:59.161850 2348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:32:59.161890 2348 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 16:32:59.161906 2348 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 16:32:59.161937 2348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 16:32:59.167254 2348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2348
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:32:59.168283 2348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:32:59.169265 2348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:32:59.178979 2356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:32:59.178972 2354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:32:59.178972 2353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:32:59.179231 2348 server_base.cc:1034] running on GCE node
I20260925 16:32:59.179750 2348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:32:59.180408 2348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:32:59.181594 2348 hybrid_clock.cc:648] HybridClock initialized: now 1790353979181577 us; error 33 us; skew 500 ppm
Sep 25 16:32:59 dist-test-slave-65pf krb5kdc[2091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353979, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:32:59.184394 2348 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 16:32:59.185331 2348 webserver.cc:460] Webserver started at http://127.0.98.2:35139/ using document root <none> and password file <none>
I20260925 16:32:59.185910 2348 fs_manager.cc:359] Metadata directory not provided
I20260925 16:32:59.186002 2348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:32:59.186244 2348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:32:59.188438 2348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/ts-1/data/instance:
uuid: "97509c025dbe428fabba21071efa72ce"
format_stamp: "Formatted at 2026-09-25 16:32:59 on dist-test-slave-65pf"
server_key: "ec547bd6fa867e3e4fad7b2a600abaff"
server_key_iv: "53a89958462d0c97c5744dc94c5020ef"
server_key_version: "encryptionkey@0"
I20260925 16:32:59.189019 2348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/ts-1/wal/instance:
uuid: "97509c025dbe428fabba21071efa72ce"
format_stamp: "Formatted at 2026-09-25 16:32:59 on dist-test-slave-65pf"
server_key: "ec547bd6fa867e3e4fad7b2a600abaff"
server_key_iv: "53a89958462d0c97c5744dc94c5020ef"
server_key_version: "encryptionkey@0"
I20260925 16:32:59.192934 2348 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 16:32:59.195469 2363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:32:59.196712 2348 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:32:59.196849 2348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/ts-1/wal
uuid: "97509c025dbe428fabba21071efa72ce"
format_stamp: "Formatted at 2026-09-25 16:32:59 on dist-test-slave-65pf"
server_key: "ec547bd6fa867e3e4fad7b2a600abaff"
server_key_iv: "53a89958462d0c97c5744dc94c5020ef"
server_key_version: "encryptionkey@0"
I20260925 16:32:59.196957 2348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:32:59.220125 2348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:32:59.223227 2348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:32:59.223434 2348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:32:59.224912 2348 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:32:59.224988 2348 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:32:59.225059 2348 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:32:59.225116 2348 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 16:32:59 dist-test-slave-65pf krb5kdc[2091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353979, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:32:59.235652 2348 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37461
I20260925 16:32:59.235677 2478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37461 every 8 connection(s)
I20260925 16:32:59.236545 2348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
I20260925 16:32:59.238883 2185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:59.225689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60525 (local address 127.0.98.62:45895)
0925 16:32:59.225847 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:59.225853 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:59.226702 (+ 849us) server_negotiation.cc:407] Connection header received
0925 16:32:59.227672 (+ 970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:59.227676 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:59.227731 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:59.227823 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:59.229490 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:59.230174 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:59.232090 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:59.232340 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:59.232464 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:59.235133 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:32:59.235136 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:32:59.235139 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:32:59.235173 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:32:59.236814 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:59.237394 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:59.237398 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:59.237400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:59.237493 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:59.237844 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:59.237846 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:59.237847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:59.238022 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:32:59.238097 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:59.238647 (+ 550us) server_negotiation.cc:299] Negotiation successful
0925 16:32:59.238767 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 16:32:59.239629 2372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:59.226072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45895 (local address 127.0.98.2:60525)
0925 16:32:59.226519 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:59.226570 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 16:32:59.227458 (+ 888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:32:59.228062 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:32:59.228072 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:32:59.228615 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:32:59.229327 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:59.229353 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:59.230280 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:59.230283 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:59.231963 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:59.231979 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:59.232528 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:59.232534 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:59.232716 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:59.233343 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:32:59.233359 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:32:59.234956 (+ 1597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:32:59.236939 (+ 1983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:59.236944 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:59.236959 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:59.237289 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:59.237624 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:59.237626 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:59.237627 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:59.237752 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:59.238110 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:32:59.238115 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:32:59.238331 (+ 216us) client_negotiation.cc:769] Sending connection context
0925 16:32:59.238481 (+ 150us) client_negotiation.cc:240] Negotiation successful
0925 16:32:59.238795 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":130,"threads_started":1}
I20260925 16:32:59.240808 2479 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45895
I20260925 16:32:59.240960 2479 heartbeater.cc:464] Registering TS with master...
I20260925 16:32:59.241479 2479 heartbeater.cc:511] Master 127.0.98.62:45895 requested a full tablet report, sending...
I20260925 16:32:59.242342 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 2348
I20260925 16:32:59.242453 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/ts-1/wal/instance
I20260925 16:32:59.242504 2136 ts_manager.cc:194] Registered new tserver with Master: 97509c025dbe428fabba21071efa72ce (127.0.98.2:37461)
I20260925 16:32:59.242779 392 external_mini_cluster.cc:1428] Setting key c67e51fcd0ac5414658751004a2090d5
I20260925 16:32:59.243175 2136 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60525
I20260925 16:32:59.251073 2485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:59.243369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45895 (local address 127.0.98.2:41143)
0925 16:32:59.243643 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:59.243656 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:32:59.243760 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:32:59.244229 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:32:59.244232 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:32:59.244434 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:32:59.244525 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:59.244531 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:59.245260 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:59.245263 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:59.245974 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:59.245987 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:59.246426 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:59.246431 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:59.246545 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:59.247097 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:32:59.247116 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:32:59.247458 (+ 342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:32:59.249446 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:59.249449 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:59.249450 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:59.249664 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:59.249978 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:59.249980 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:59.249981 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:59.250062 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:59.250641 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:32:59.250644 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:32:59.250736 (+ 92us) client_negotiation.cc:769] Sending connection context
0925 16:32:59.250820 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 16:32:59.250927 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":113,"threads_started":1}
I20260925 16:32:59.251253 2185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:59.243893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41143 (local address 127.0.98.62:45895)
0925 16:32:59.244025 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:59.244029 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:59.244041 (+ 12us) server_negotiation.cc:407] Connection header received
0925 16:32:59.244075 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:59.244077 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:59.244124 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:59.244211 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:59.244636 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:59.245138 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:59.246101 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:59.246310 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:59.246394 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:59.247593 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:32:59.247600 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:32:59.247606 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:32:59.247637 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:32:59.249322 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:59.249761 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:59.249768 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:59.249772 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:59.249876 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:59.250159 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:59.250162 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:59.250164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:59.250482 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:32:59.250610 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:59.250927 (+ 317us) server_negotiation.cc:299] Negotiation successful
0925 16:32:59.251070 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 16:32:59.271426 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45895
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:32:59.422423 2488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:32:59.422775 2488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:32:59.422861 2488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:32:59.426982 2488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:32:59.427091 2488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:32:59.427160 2488 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 16:32:59.427191 2488 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 16:32:59.427245 2488 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 16:32:59.432547 2488 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2488
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:32:59.433610 2488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:32:59.434701 2488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:32:59.444331 2494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:32:59.444531 2488 server_base.cc:1034] running on GCE node
W20260925 16:32:59.444340 2493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:32:59.444684 2496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:32:59.445237 2488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:32:59.446108 2488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:32:59.447321 2488 hybrid_clock.cc:648] HybridClock initialized: now 1790353979447285 us; error 44 us; skew 500 ppm
Sep 25 16:32:59 dist-test-slave-65pf krb5kdc[2091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353979, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:32:59.451107 2488 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 16:32:59.452160 2488 webserver.cc:460] Webserver started at http://127.0.98.3:46715/ using document root <none> and password file <none>
I20260925 16:32:59.452777 2488 fs_manager.cc:359] Metadata directory not provided
I20260925 16:32:59.452870 2488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:32:59.453112 2488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:32:59.455307 2488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/ts-2/data/instance:
uuid: "a3d3aa90f8e74bb2ae128513c9f0523a"
format_stamp: "Formatted at 2026-09-25 16:32:59 on dist-test-slave-65pf"
server_key: "4d9509e45fce297a81981cb4c001fe29"
server_key_iv: "3d16ff13966d12d2b69c321b1aee447f"
server_key_version: "encryptionkey@0"
I20260925 16:32:59.455886 2488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a3d3aa90f8e74bb2ae128513c9f0523a"
format_stamp: "Formatted at 2026-09-25 16:32:59 on dist-test-slave-65pf"
server_key: "4d9509e45fce297a81981cb4c001fe29"
server_key_iv: "3d16ff13966d12d2b69c321b1aee447f"
server_key_version: "encryptionkey@0"
I20260925 16:32:59.459882 2488 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260925 16:32:59.462376 2503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:32:59.463718 2488 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:32:59.463840 2488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/ts-2/wal
uuid: "a3d3aa90f8e74bb2ae128513c9f0523a"
format_stamp: "Formatted at 2026-09-25 16:32:59 on dist-test-slave-65pf"
server_key: "4d9509e45fce297a81981cb4c001fe29"
server_key_iv: "3d16ff13966d12d2b69c321b1aee447f"
server_key_version: "encryptionkey@0"
I20260925 16:32:59.463941 2488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:32:59.483119 2488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:32:59.486310 2488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:32:59.486509 2488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:32:59.488353 2488 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:32:59.488454 2488 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:32:59.488554 2488 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:32:59.488588 2488 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:32:59.498759 2488 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43817
I20260925 16:32:59.499060 2618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43817 every 8 connection(s)
Sep 25 16:32:59 dist-test-slave-65pf krb5kdc[2091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353979, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:32:59.499959 2488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
I20260925 16:32:59.504599 2185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:59.491328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57821 (local address 127.0.98.62:45895)
0925 16:32:59.491472 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:59.491479 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:59.491500 (+ 21us) server_negotiation.cc:407] Connection header received
0925 16:32:59.491797 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:59.491802 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:59.491868 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:59.491935 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:59.493843 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:59.494757 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:59.496519 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:59.496988 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:59.497130 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:59.500551 (+ 3421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:32:59.500557 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:32:59.500561 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:32:59.500595 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:32:59.502330 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:59.503029 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:59.503032 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:59.503033 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:59.503104 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:59.503520 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:59.503522 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:59.503523 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:59.503685 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:32:59.503776 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:59.504380 (+ 604us) server_negotiation.cc:299] Negotiation successful
0925 16:32:59.504475 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 16:32:59.505482 2512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:59.489642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45895 (local address 127.0.98.3:57821)
0925 16:32:59.490219 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:59.490277 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 16:32:59.491501 (+ 1224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:32:59.492355 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:32:59.492363 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:32:59.492899 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:32:59.493620 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:59.493657 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:59.494919 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:59.494924 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:59.496378 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:59.496390 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:59.497173 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:59.497181 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:59.497495 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:59.498372 (+ 877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:32:59.498398 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:32:59.500341 (+ 1943us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:32:59.502433 (+ 2092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:59.502442 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:59.502460 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:59.502917 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:59.503199 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:59.503204 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:59.503207 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:59.503418 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:59.503802 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:32:59.503809 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:32:59.504115 (+ 306us) client_negotiation.cc:769] Sending connection context
0925 16:32:59.504308 (+ 193us) client_negotiation.cc:240] Negotiation successful
0925 16:32:59.504648 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":143,"threads_started":1}
I20260925 16:32:59.506827 2619 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45895
I20260925 16:32:59.507007 2619 heartbeater.cc:464] Registering TS with master...
I20260925 16:32:59.507534 2619 heartbeater.cc:511] Master 127.0.98.62:45895 requested a full tablet report, sending...
I20260925 16:32:59.508671 2136 ts_manager.cc:194] Registered new tserver with Master: a3d3aa90f8e74bb2ae128513c9f0523a (127.0.98.3:43817)
I20260925 16:32:59.509526 2136 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57821
I20260925 16:32:59.509681 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 2488
I20260925 16:32:59.509817 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/ts-2/wal/instance
I20260925 16:32:59.510139 392 external_mini_cluster.cc:1428] Setting key 67bf23ce75e40350abb2369eea2bd403
I20260925 16:32:59.512143 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 16:32:59.516928 2185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:59.509039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40327 (local address 127.0.98.62:45895)
0925 16:32:59.509164 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:59.509167 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:59.509302 (+ 135us) server_negotiation.cc:407] Connection header received
0925 16:32:59.509453 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:59.509457 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:59.509502 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:59.509589 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:59.510388 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:59.511108 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:59.512138 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:59.512381 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:59.512510 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:59.513702 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:32:59.513706 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:32:59.513709 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:32:59.513733 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:32:59.515263 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:59.515745 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:59.515749 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:59.515750 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:59.515862 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:59.516131 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:59.516135 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:59.516137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:59.516317 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:32:59.516418 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:59.516677 (+ 259us) server_negotiation.cc:299] Negotiation successful
0925 16:32:59.516780 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 16:32:59.517074 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:59.508920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45895 (local address 127.0.98.3:40327)
0925 16:32:59.509223 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:59.509240 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 16:32:59.509346 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:32:59.509689 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:32:59.509693 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:32:59.510112 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:32:59.510246 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:59.510255 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:59.511236 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:59.511240 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:59.512005 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:59.512014 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:59.512514 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:59.512518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:59.512622 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:59.513207 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:32:59.513220 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:32:59.513547 (+ 327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:32:59.515397 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:59.515400 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:59.515402 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:59.515610 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:59.515940 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:59.515945 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:59.515947 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:59.516037 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:59.516418 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:32:59.516421 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:32:59.516514 (+ 93us) client_negotiation.cc:769] Sending connection context
0925 16:32:59.516629 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 16:32:59.516771 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":93,"threads_started":1}
Sep 25 16:32:59 dist-test-slave-65pf krb5kdc[2091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 16:32:59 dist-test-slave-65pf krb5kdc[2091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:32:59.537405 2185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:59.529759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55664 (local address 127.0.98.62:45895)
0925 16:32:59.529885 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:59.529888 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:59.529900 (+ 12us) server_negotiation.cc:407] Connection header received
0925 16:32:59.529937 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:59.529939 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:59.529975 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:59.530045 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:59.530749 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:59.531280 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:59.532091 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:59.532320 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:59.532399 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:59.534570 (+ 2171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:32:59.534573 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:32:59.534575 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:32:59.534676 (+ 101us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:32:59.535934 (+ 1258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:59.536311 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:59.536314 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:59.536315 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:59.536394 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:59.536618 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:59.536620 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:59.536621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:59.536786 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:32:59.536891 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:59.537068 (+ 177us) server_negotiation.cc:299] Negotiation successful
0925 16:32:59.537201 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 16:32:59.540576 2136 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55664:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 16:32:59.543658 2136 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:32:59.558132 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:59.552724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37916 (local address 127.0.98.2:37461)
0925 16:32:59.553185 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:59.553198 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:59.553413 (+ 215us) server_negotiation.cc:407] Connection header received
0925 16:32:59.553548 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:59.553553 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:59.553741 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:59.553882 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:32:59.554097 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:59.555017 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:59.556647 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:59.557449 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:59.557561 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:59.557738 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:59.557873 (+ 135us) server_negotiation.cc:299] Negotiation successful
0925 16:32:59.557970 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":98,"threads_started":1}
I20260925 16:32:59.558132 2638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:59.552334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37461 (local address 127.0.0.1:37916)
0925 16:32:59.553317 (+ 983us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:59.553327 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 16:32:59.553441 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:32:59.553870 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:32:59.553873 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:32:59.553888 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:32:59.553996 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:59.554001 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:59.555131 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:59.555134 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:59.556518 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:59.556527 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:59.557574 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:59.557578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:59.557678 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:59.557738 (+ 60us) client_negotiation.cc:769] Sending connection context
0925 16:32:59.557847 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 16:32:59.557971 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":934,"mutex_wait_us":167,"thread_start_us":80,"threads_started":1}
I20260925 16:32:59.558508 2640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:59.552544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41490 (local address 127.0.98.3:43817)
0925 16:32:59.552946 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:59.552958 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:59.553182 (+ 224us) server_negotiation.cc:407] Connection header received
0925 16:32:59.553344 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:59.553349 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:59.553513 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:59.553673 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:32:59.554002 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:59.555164 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:59.556880 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:59.557664 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:59.557768 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:59.557910 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:59.558252 (+ 342us) server_negotiation.cc:299] Negotiation successful
0925 16:32:59.558365 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":102,"threads_started":1}
I20260925 16:32:59.558857 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:59.551923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43805 (local address 127.0.0.1:58388)
0925 16:32:59.552745 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:59.552797 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 16:32:59.552926 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:32:59.554198 (+ 1272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:32:59.554202 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:32:59.554218 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:32:59.554308 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:59.554314 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:59.555718 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:59.555721 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:59.556928 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:59.556935 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:59.557973 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:59.557976 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:59.558081 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:59.558103 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 16:32:59.558172 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 16:32:59.558708 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"mutex_wait_us":99,"thread_start_us":85,"threads_started":1}
I20260925 16:32:59.559212 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:59.552076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58388 (local address 127.0.98.1:43805)
0925 16:32:59.553666 (+ 1590us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:59.553672 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:59.553691 (+ 19us) server_negotiation.cc:407] Connection header received
0925 16:32:59.553759 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:59.553764 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:59.553921 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:59.554048 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:32:59.554436 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:59.555580 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:59.557059 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:59.557660 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:59.557761 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:59.557914 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:59.558507 (+ 593us) server_negotiation.cc:299] Negotiation successful
0925 16:32:59.559071 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1496,"thread_start_us":85,"threads_started":1}
I20260925 16:32:59.559706 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:59.552072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43817 (local address 127.0.0.1:41490)
0925 16:32:59.553088 (+ 1016us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:59.553099 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 16:32:59.553199 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:32:59.553661 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:32:59.553667 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:32:59.553688 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:32:59.553870 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:59.553877 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:59.555314 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:59.555317 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:59.556738 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:59.556748 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:59.557967 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:59.557970 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:59.558071 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:59.558099 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 16:32:59.559432 (+ 1333us) client_negotiation.cc:240] Negotiation successful
0925 16:32:59.559566 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":954,"mutex_wait_us":47,"thread_start_us":88,"threads_started":1}
I20260925 16:32:59.561717 2413 tablet_service.cc:1467] Processing CreateTablet for tablet 0c4e5abf31d743958b58b4089a56ec9f (DEFAULT_TABLE table=test-table [id=a7eab0096a42478c9af60eb344c780a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:32:59.561705 2273 tablet_service.cc:1467] Processing CreateTablet for tablet 0c4e5abf31d743958b58b4089a56ec9f (DEFAULT_TABLE table=test-table [id=a7eab0096a42478c9af60eb344c780a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:32:59.561911 2553 tablet_service.cc:1467] Processing CreateTablet for tablet 0c4e5abf31d743958b58b4089a56ec9f (DEFAULT_TABLE table=test-table [id=a7eab0096a42478c9af60eb344c780a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:32:59.562837 2413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c4e5abf31d743958b58b4089a56ec9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:32:59.562841 2273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c4e5abf31d743958b58b4089a56ec9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:32:59.562868 2553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c4e5abf31d743958b58b4089a56ec9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:32:59.569391 2642 tablet_bootstrap.cc:492] T 0c4e5abf31d743958b58b4089a56ec9f P a3d3aa90f8e74bb2ae128513c9f0523a: Bootstrap starting.
I20260925 16:32:59.570672 2644 tablet_bootstrap.cc:492] T 0c4e5abf31d743958b58b4089a56ec9f P 97509c025dbe428fabba21071efa72ce: Bootstrap starting.
I20260925 16:32:59.570905 2643 tablet_bootstrap.cc:492] T 0c4e5abf31d743958b58b4089a56ec9f P 435b9fb0fe93430face8f32e4de081b4: Bootstrap starting.
I20260925 16:32:59.571924 2642 tablet_bootstrap.cc:654] T 0c4e5abf31d743958b58b4089a56ec9f P a3d3aa90f8e74bb2ae128513c9f0523a: Neither blocks nor log segments found. Creating new log.
I20260925 16:32:59.572706 2644 tablet_bootstrap.cc:654] T 0c4e5abf31d743958b58b4089a56ec9f P 97509c025dbe428fabba21071efa72ce: Neither blocks nor log segments found. Creating new log.
I20260925 16:32:59.572779 2642 log.cc:824] T 0c4e5abf31d743958b58b4089a56ec9f P a3d3aa90f8e74bb2ae128513c9f0523a: Log is configured to *not* fsync() on all Append() calls
I20260925 16:32:59.572824 2643 tablet_bootstrap.cc:654] T 0c4e5abf31d743958b58b4089a56ec9f P 435b9fb0fe93430face8f32e4de081b4: Neither blocks nor log segments found. Creating new log.
I20260925 16:32:59.573482 2644 log.cc:824] T 0c4e5abf31d743958b58b4089a56ec9f P 97509c025dbe428fabba21071efa72ce: Log is configured to *not* fsync() on all Append() calls
I20260925 16:32:59.574049 2643 log.cc:824] T 0c4e5abf31d743958b58b4089a56ec9f P 435b9fb0fe93430face8f32e4de081b4: Log is configured to *not* fsync() on all Append() calls
I20260925 16:32:59.575130 2642 tablet_bootstrap.cc:492] T 0c4e5abf31d743958b58b4089a56ec9f P a3d3aa90f8e74bb2ae128513c9f0523a: No bootstrap required, opened a new log
I20260925 16:32:59.575311 2644 tablet_bootstrap.cc:492] T 0c4e5abf31d743958b58b4089a56ec9f P 97509c025dbe428fabba21071efa72ce: No bootstrap required, opened a new log
I20260925 16:32:59.575327 2642 ts_tablet_manager.cc:1397] T 0c4e5abf31d743958b58b4089a56ec9f P a3d3aa90f8e74bb2ae128513c9f0523a: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 16:32:59.575498 2644 ts_tablet_manager.cc:1397] T 0c4e5abf31d743958b58b4089a56ec9f P 97509c025dbe428fabba21071efa72ce: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 16:32:59.575845 2643 tablet_bootstrap.cc:492] T 0c4e5abf31d743958b58b4089a56ec9f P 435b9fb0fe93430face8f32e4de081b4: No bootstrap required, opened a new log
I20260925 16:32:59.576017 2643 ts_tablet_manager.cc:1397] T 0c4e5abf31d743958b58b4089a56ec9f P 435b9fb0fe93430face8f32e4de081b4: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 16:32:59.578380 2644 raft_consensus.cc:348] T 0c4e5abf31d743958b58b4089a56ec9f P 97509c025dbe428fabba21071efa72ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435b9fb0fe93430face8f32e4de081b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43805 } } peers { permanent_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43817 } } peers { permanent_uuid: "97509c025dbe428fabba21071efa72ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37461 } }
I20260925 16:32:59.578647 2644 raft_consensus.cc:374] T 0c4e5abf31d743958b58b4089a56ec9f P 97509c025dbe428fabba21071efa72ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:32:59.578702 2644 raft_consensus.cc:729] T 0c4e5abf31d743958b58b4089a56ec9f P 97509c025dbe428fabba21071efa72ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97509c025dbe428fabba21071efa72ce, State: Initialized, Role: FOLLOWER
I20260925 16:32:59.579149 2644 consensus_queue.cc:260] T 0c4e5abf31d743958b58b4089a56ec9f P 97509c025dbe428fabba21071efa72ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435b9fb0fe93430face8f32e4de081b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43805 } } peers { permanent_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43817 } } peers { permanent_uuid: "97509c025dbe428fabba21071efa72ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37461 } }
I20260925 16:32:59.579263 2642 raft_consensus.cc:348] T 0c4e5abf31d743958b58b4089a56ec9f P a3d3aa90f8e74bb2ae128513c9f0523a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435b9fb0fe93430face8f32e4de081b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43805 } } peers { permanent_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43817 } } peers { permanent_uuid: "97509c025dbe428fabba21071efa72ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37461 } }
I20260925 16:32:59.579509 2642 raft_consensus.cc:374] T 0c4e5abf31d743958b58b4089a56ec9f P a3d3aa90f8e74bb2ae128513c9f0523a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:32:59.579568 2642 raft_consensus.cc:729] T 0c4e5abf31d743958b58b4089a56ec9f P a3d3aa90f8e74bb2ae128513c9f0523a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3d3aa90f8e74bb2ae128513c9f0523a, State: Initialized, Role: FOLLOWER
I20260925 16:32:59.579879 2479 heartbeater.cc:503] Master 127.0.98.62:45895 was elected leader, sending a full tablet report...
I20260925 16:32:59.579738 2643 raft_consensus.cc:348] T 0c4e5abf31d743958b58b4089a56ec9f P 435b9fb0fe93430face8f32e4de081b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435b9fb0fe93430face8f32e4de081b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43805 } } peers { permanent_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43817 } } peers { permanent_uuid: "97509c025dbe428fabba21071efa72ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37461 } }
I20260925 16:32:59.579983 2643 raft_consensus.cc:374] T 0c4e5abf31d743958b58b4089a56ec9f P 435b9fb0fe93430face8f32e4de081b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:32:59.580031 2643 raft_consensus.cc:729] T 0c4e5abf31d743958b58b4089a56ec9f P 435b9fb0fe93430face8f32e4de081b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 435b9fb0fe93430face8f32e4de081b4, State: Initialized, Role: FOLLOWER
I20260925 16:32:59.580408 2643 consensus_queue.cc:260] T 0c4e5abf31d743958b58b4089a56ec9f P 435b9fb0fe93430face8f32e4de081b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435b9fb0fe93430face8f32e4de081b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43805 } } peers { permanent_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43817 } } peers { permanent_uuid: "97509c025dbe428fabba21071efa72ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37461 } }
I20260925 16:32:59.580523 2642 consensus_queue.cc:260] T 0c4e5abf31d743958b58b4089a56ec9f P a3d3aa90f8e74bb2ae128513c9f0523a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435b9fb0fe93430face8f32e4de081b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43805 } } peers { permanent_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43817 } } peers { permanent_uuid: "97509c025dbe428fabba21071efa72ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37461 } }
I20260925 16:32:59.581058 2339 heartbeater.cc:503] Master 127.0.98.62:45895 was elected leader, sending a full tablet report...
I20260925 16:32:59.581389 2643 ts_tablet_manager.cc:1428] T 0c4e5abf31d743958b58b4089a56ec9f P 435b9fb0fe93430face8f32e4de081b4: Time spent starting tablet: real 0.005s user 0.000s sys 0.006s
I20260925 16:32:59.581482 2644 ts_tablet_manager.cc:1428] T 0c4e5abf31d743958b58b4089a56ec9f P 97509c025dbe428fabba21071efa72ce: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260925 16:32:59.581770 2619 heartbeater.cc:503] Master 127.0.98.62:45895 was elected leader, sending a full tablet report...
I20260925 16:32:59.582491 2642 ts_tablet_manager.cc:1428] T 0c4e5abf31d743958b58b4089a56ec9f P a3d3aa90f8e74bb2ae128513c9f0523a: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260925 16:32:59.738025 2480 tablet.cc:2369] T 0c4e5abf31d743958b58b4089a56ec9f P 97509c025dbe428fabba21071efa72ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 16:32:59.751192 2620 tablet.cc:2369] T 0c4e5abf31d743958b58b4089a56ec9f P a3d3aa90f8e74bb2ae128513c9f0523a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:32:59.763954 2650 raft_consensus.cc:482] T 0c4e5abf31d743958b58b4089a56ec9f P a3d3aa90f8e74bb2ae128513c9f0523a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:32:59.764245 2650 raft_consensus.cc:504] T 0c4e5abf31d743958b58b4089a56ec9f P a3d3aa90f8e74bb2ae128513c9f0523a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435b9fb0fe93430face8f32e4de081b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43805 } } peers { permanent_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43817 } } peers { permanent_uuid: "97509c025dbe428fabba21071efa72ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37461 } }
W20260925 16:32:59.764999 2340 tablet.cc:2369] T 0c4e5abf31d743958b58b4089a56ec9f P 435b9fb0fe93430face8f32e4de081b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:32:59.765709 2650 leader_election.cc:288] T 0c4e5abf31d743958b58b4089a56ec9f P a3d3aa90f8e74bb2ae128513c9f0523a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 435b9fb0fe93430face8f32e4de081b4 (127.0.98.1:43805), 97509c025dbe428fabba21071efa72ce (127.0.98.2:37461)
I20260925 16:32:59.770308 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:59.766084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52893 (local address 127.0.98.1:43805)
0925 16:32:59.766215 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:59.766220 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:59.766241 (+ 21us) server_negotiation.cc:407] Connection header received
0925 16:32:59.766329 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:59.766334 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:59.766391 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:59.766503 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:32:59.766849 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:59.767745 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:59.769130 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:59.769737 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:59.769855 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:59.769936 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:59.770152 (+ 216us) server_negotiation.cc:299] Negotiation successful
0925 16:32:59.770210 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 16:32:59.770329 2512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:59.765990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43805 (local address 127.0.98.3:52893)
0925 16:32:59.766097 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:59.766115 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 16:32:59.766215 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:32:59.766510 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:32:59.766513 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:32:59.766554 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:32:59.766692 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:59.766697 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:59.767868 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:59.767872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:59.769008 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:59.769017 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:59.769929 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:59.769932 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:59.770031 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:59.770056 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 16:32:59.770133 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 16:32:59.770199 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 16:32:59.771097 2651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:59.765990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37461 (local address 127.0.98.3:53751)
0925 16:32:59.766451 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:59.766468 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 16:32:59.766648 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:32:59.766943 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:32:59.766947 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:32:59.766965 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:32:59.767576 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:59.767585 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:59.768668 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:59.768671 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:59.769720 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:59.769731 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:59.770615 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:59.770619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:59.770720 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:59.770741 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 16:32:59.770887 (+ 146us) client_negotiation.cc:240] Negotiation successful
0925 16:32:59.770968 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"mutex_wait_us":122,"spinlock_wait_cycles":92544,"thread_start_us":131,"threads_started":1}
I20260925 16:32:59.771106 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:59.766084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53751 (local address 127.0.98.2:37461)
0925 16:32:59.766215 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:59.766220 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:59.766562 (+ 342us) server_negotiation.cc:407] Connection header received
0925 16:32:59.766780 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:59.766786 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:59.766849 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:59.766917 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:32:59.767716 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:59.768503 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:59.769879 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:59.770451 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:59.770559 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:59.770646 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:59.770928 (+ 282us) server_negotiation.cc:299] Negotiation successful
0925 16:32:59.770998 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 16:32:59.771095 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c4e5abf31d743958b58b4089a56ec9f" candidate_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "435b9fb0fe93430face8f32e4de081b4" is_pre_election: true
I20260925 16:32:59.771432 2293 raft_consensus.cc:2454] T 0c4e5abf31d743958b58b4089a56ec9f P 435b9fb0fe93430face8f32e4de081b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3d3aa90f8e74bb2ae128513c9f0523a in term 0.
I20260925 16:32:59.771746 2433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c4e5abf31d743958b58b4089a56ec9f" candidate_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97509c025dbe428fabba21071efa72ce" is_pre_election: true
I20260925 16:32:59.771960 2505 leader_election.cc:302] T 0c4e5abf31d743958b58b4089a56ec9f P a3d3aa90f8e74bb2ae128513c9f0523a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 435b9fb0fe93430face8f32e4de081b4, a3d3aa90f8e74bb2ae128513c9f0523a; no voters:
I20260925 16:32:59.772002 2433 raft_consensus.cc:2454] T 0c4e5abf31d743958b58b4089a56ec9f P 97509c025dbe428fabba21071efa72ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3d3aa90f8e74bb2ae128513c9f0523a in term 0.
I20260925 16:32:59.772240 2650 raft_consensus.cc:2781] T 0c4e5abf31d743958b58b4089a56ec9f P a3d3aa90f8e74bb2ae128513c9f0523a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:32:59.772353 2650 raft_consensus.cc:482] T 0c4e5abf31d743958b58b4089a56ec9f P a3d3aa90f8e74bb2ae128513c9f0523a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:32:59.772408 2650 raft_consensus.cc:3037] T 0c4e5abf31d743958b58b4089a56ec9f P a3d3aa90f8e74bb2ae128513c9f0523a [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:32:59.773437 2650 raft_consensus.cc:504] T 0c4e5abf31d743958b58b4089a56ec9f P a3d3aa90f8e74bb2ae128513c9f0523a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435b9fb0fe93430face8f32e4de081b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43805 } } peers { permanent_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43817 } } peers { permanent_uuid: "97509c025dbe428fabba21071efa72ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37461 } }
I20260925 16:32:59.773747 2650 leader_election.cc:288] T 0c4e5abf31d743958b58b4089a56ec9f P a3d3aa90f8e74bb2ae128513c9f0523a [CANDIDATE]: Term 1 election: Requested vote from peers 435b9fb0fe93430face8f32e4de081b4 (127.0.98.1:43805), 97509c025dbe428fabba21071efa72ce (127.0.98.2:37461)
I20260925 16:32:59.774065 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c4e5abf31d743958b58b4089a56ec9f" candidate_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "435b9fb0fe93430face8f32e4de081b4"
I20260925 16:32:59.774056 2433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c4e5abf31d743958b58b4089a56ec9f" candidate_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97509c025dbe428fabba21071efa72ce"
I20260925 16:32:59.774186 2293 raft_consensus.cc:3037] T 0c4e5abf31d743958b58b4089a56ec9f P 435b9fb0fe93430face8f32e4de081b4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:32:59.774186 2433 raft_consensus.cc:3037] T 0c4e5abf31d743958b58b4089a56ec9f P 97509c025dbe428fabba21071efa72ce [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:32:59.775450 2433 raft_consensus.cc:2454] T 0c4e5abf31d743958b58b4089a56ec9f P 97509c025dbe428fabba21071efa72ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3d3aa90f8e74bb2ae128513c9f0523a in term 1.
I20260925 16:32:59.775450 2293 raft_consensus.cc:2454] T 0c4e5abf31d743958b58b4089a56ec9f P 435b9fb0fe93430face8f32e4de081b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3d3aa90f8e74bb2ae128513c9f0523a in term 1.
I20260925 16:32:59.775750 2507 leader_election.cc:302] T 0c4e5abf31d743958b58b4089a56ec9f P a3d3aa90f8e74bb2ae128513c9f0523a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97509c025dbe428fabba21071efa72ce, a3d3aa90f8e74bb2ae128513c9f0523a; no voters:
I20260925 16:32:59.775949 2650 raft_consensus.cc:2781] T 0c4e5abf31d743958b58b4089a56ec9f P a3d3aa90f8e74bb2ae128513c9f0523a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:32:59.776162 2650 raft_consensus.cc:686] T 0c4e5abf31d743958b58b4089a56ec9f P a3d3aa90f8e74bb2ae128513c9f0523a [term 1 LEADER]: Becoming Leader. State: Replica: a3d3aa90f8e74bb2ae128513c9f0523a, State: Running, Role: LEADER
I20260925 16:32:59.776466 2650 consensus_queue.cc:237] T 0c4e5abf31d743958b58b4089a56ec9f P a3d3aa90f8e74bb2ae128513c9f0523a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435b9fb0fe93430face8f32e4de081b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43805 } } peers { permanent_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43817 } } peers { permanent_uuid: "97509c025dbe428fabba21071efa72ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37461 } }
I20260925 16:32:59.779486 2135 catalog_manager.cc:5495] T 0c4e5abf31d743958b58b4089a56ec9f P a3d3aa90f8e74bb2ae128513c9f0523a reported cstate change: term changed from 0 to 1, leader changed from <none> to a3d3aa90f8e74bb2ae128513c9f0523a (127.0.98.3). New cstate: current_term: 1 leader_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435b9fb0fe93430face8f32e4de081b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97509c025dbe428fabba21071efa72ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37461 } health_report { overall_health: UNKNOWN } } }
Sep 25 16:32:59 dist-test-slave-65pf krb5kdc[2091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353978, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:32:59.820911 2185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:59.812306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55676 (local address 127.0.98.62:45895)
0925 16:32:59.812457 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:59.812463 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:59.812678 (+ 215us) server_negotiation.cc:407] Connection header received
0925 16:32:59.812832 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:59.812838 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:59.812894 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:59.812977 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:59.813641 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:59.814151 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:59.815011 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:59.815216 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:59.815327 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:59.817695 (+ 2368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:32:59.817698 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:32:59.817701 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:32:59.817729 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:32:59.819210 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:59.819659 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:59.819665 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:59.819671 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:59.819734 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:59.819990 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:59.819996 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:59.819998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:59.820169 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:32:59.820268 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:59.820560 (+ 292us) server_negotiation.cc:299] Negotiation successful
0925 16:32:59.820752 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":96,"server-negotiator.queue_time_us":55}
I20260925 16:32:59.820919 2659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:59.812263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45895 (local address 127.0.0.1:55676)
0925 16:32:59.812583 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:59.812596 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:32:59.812735 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:32:59.812995 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:32:59.812997 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:32:59.813412 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:32:59.813522 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:59.813528 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:59.814259 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:59.814265 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:59.814900 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:59.814909 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:59.815320 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:59.815323 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:59.815395 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:59.816020 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:32:59.816031 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:32:59.817547 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:32:59.819314 (+ 1767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:59.819325 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:59.819331 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:59.819547 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:59.819824 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:59.819827 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:59.819829 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:59.819906 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:59.820277 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:32:59.820286 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:32:59.820398 (+ 112us) client_negotiation.cc:769] Sending connection context
0925 16:32:59.820509 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 16:32:59.820679 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":140,"threads_started":1}
I20260925 16:32:59.824769 2135 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55676:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 16:32:59.825542 2135 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:32:59.830559 2553 tablet_service.cc:1467] Processing CreateTablet for tablet 4ff1cb81a17041e18460ada65b07f1cc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ec385253c2c2404fb18151cabb129cad]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 16:32:59.830552 2273 tablet_service.cc:1467] Processing CreateTablet for tablet 4ff1cb81a17041e18460ada65b07f1cc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ec385253c2c2404fb18151cabb129cad]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 16:32:59.830544 2413 tablet_service.cc:1467] Processing CreateTablet for tablet 4ff1cb81a17041e18460ada65b07f1cc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ec385253c2c2404fb18151cabb129cad]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 16:32:59.831357 2553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ff1cb81a17041e18460ada65b07f1cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:32:59.831357 2273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ff1cb81a17041e18460ada65b07f1cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:32:59.831370 2413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ff1cb81a17041e18460ada65b07f1cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:32:59.833729 2644 tablet_bootstrap.cc:492] T 4ff1cb81a17041e18460ada65b07f1cc P 97509c025dbe428fabba21071efa72ce: Bootstrap starting.
I20260925 16:32:59.834159 2643 tablet_bootstrap.cc:492] T 4ff1cb81a17041e18460ada65b07f1cc P 435b9fb0fe93430face8f32e4de081b4: Bootstrap starting.
I20260925 16:32:59.834724 2644 tablet_bootstrap.cc:654] T 4ff1cb81a17041e18460ada65b07f1cc P 97509c025dbe428fabba21071efa72ce: Neither blocks nor log segments found. Creating new log.
I20260925 16:32:59.835158 2643 tablet_bootstrap.cc:654] T 4ff1cb81a17041e18460ada65b07f1cc P 435b9fb0fe93430face8f32e4de081b4: Neither blocks nor log segments found. Creating new log.
I20260925 16:32:59.835812 2642 tablet_bootstrap.cc:492] T 4ff1cb81a17041e18460ada65b07f1cc P a3d3aa90f8e74bb2ae128513c9f0523a: Bootstrap starting.
I20260925 16:32:59.836012 2644 tablet_bootstrap.cc:492] T 4ff1cb81a17041e18460ada65b07f1cc P 97509c025dbe428fabba21071efa72ce: No bootstrap required, opened a new log
I20260925 16:32:59.836098 2644 ts_tablet_manager.cc:1397] T 4ff1cb81a17041e18460ada65b07f1cc P 97509c025dbe428fabba21071efa72ce: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 16:32:59.836562 2644 raft_consensus.cc:348] T 4ff1cb81a17041e18460ada65b07f1cc P 97509c025dbe428fabba21071efa72ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435b9fb0fe93430face8f32e4de081b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43805 } } peers { permanent_uuid: "97509c025dbe428fabba21071efa72ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37461 } } peers { permanent_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43817 } }
I20260925 16:32:59.836702 2644 raft_consensus.cc:374] T 4ff1cb81a17041e18460ada65b07f1cc P 97509c025dbe428fabba21071efa72ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:32:59.836751 2642 tablet_bootstrap.cc:654] T 4ff1cb81a17041e18460ada65b07f1cc P a3d3aa90f8e74bb2ae128513c9f0523a: Neither blocks nor log segments found. Creating new log.
I20260925 16:32:59.836751 2644 raft_consensus.cc:729] T 4ff1cb81a17041e18460ada65b07f1cc P 97509c025dbe428fabba21071efa72ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97509c025dbe428fabba21071efa72ce, State: Initialized, Role: FOLLOWER
I20260925 16:32:59.836899 2644 consensus_queue.cc:260] T 4ff1cb81a17041e18460ada65b07f1cc P 97509c025dbe428fabba21071efa72ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435b9fb0fe93430face8f32e4de081b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43805 } } peers { permanent_uuid: "97509c025dbe428fabba21071efa72ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37461 } } peers { permanent_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43817 } }
I20260925 16:32:59.837177 2644 ts_tablet_manager.cc:1428] T 4ff1cb81a17041e18460ada65b07f1cc P 97509c025dbe428fabba21071efa72ce: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 16:32:59.837176 2648 tablet_replica.cc:439] T 4ff1cb81a17041e18460ada65b07f1cc P 97509c025dbe428fabba21071efa72ce: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435b9fb0fe93430face8f32e4de081b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43805 } } peers { permanent_uuid: "97509c025dbe428fabba21071efa72ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37461 } } peers { permanent_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43817 } } }
I20260925 16:32:59.837381 2648 tablet_replica.cc:442] T 4ff1cb81a17041e18460ada65b07f1cc P 97509c025dbe428fabba21071efa72ce: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 16:32:59.837759 2643 tablet_bootstrap.cc:492] T 4ff1cb81a17041e18460ada65b07f1cc P 435b9fb0fe93430face8f32e4de081b4: No bootstrap required, opened a new log
I20260925 16:32:59.837873 2643 ts_tablet_manager.cc:1397] T 4ff1cb81a17041e18460ada65b07f1cc P 435b9fb0fe93430face8f32e4de081b4: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 16:32:59.838276 2643 raft_consensus.cc:348] T 4ff1cb81a17041e18460ada65b07f1cc P 435b9fb0fe93430face8f32e4de081b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435b9fb0fe93430face8f32e4de081b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43805 } } peers { permanent_uuid: "97509c025dbe428fabba21071efa72ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37461 } } peers { permanent_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43817 } }
I20260925 16:32:59.838373 2643 raft_consensus.cc:374] T 4ff1cb81a17041e18460ada65b07f1cc P 435b9fb0fe93430face8f32e4de081b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:32:59.838397 2643 raft_consensus.cc:729] T 4ff1cb81a17041e18460ada65b07f1cc P 435b9fb0fe93430face8f32e4de081b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 435b9fb0fe93430face8f32e4de081b4, State: Initialized, Role: FOLLOWER
I20260925 16:32:59.838531 2643 consensus_queue.cc:260] T 4ff1cb81a17041e18460ada65b07f1cc P 435b9fb0fe93430face8f32e4de081b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435b9fb0fe93430face8f32e4de081b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43805 } } peers { permanent_uuid: "97509c025dbe428fabba21071efa72ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37461 } } peers { permanent_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43817 } }
I20260925 16:32:59.838838 2643 ts_tablet_manager.cc:1428] T 4ff1cb81a17041e18460ada65b07f1cc P 435b9fb0fe93430face8f32e4de081b4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 16:32:59.838845 2649 tablet_replica.cc:439] T 4ff1cb81a17041e18460ada65b07f1cc P 435b9fb0fe93430face8f32e4de081b4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435b9fb0fe93430face8f32e4de081b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43805 } } peers { permanent_uuid: "97509c025dbe428fabba21071efa72ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37461 } } peers { permanent_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43817 } } }
I20260925 16:32:59.839016 2649 tablet_replica.cc:442] T 4ff1cb81a17041e18460ada65b07f1cc P 435b9fb0fe93430face8f32e4de081b4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 16:32:59.839633 2642 tablet_bootstrap.cc:492] T 4ff1cb81a17041e18460ada65b07f1cc P a3d3aa90f8e74bb2ae128513c9f0523a: No bootstrap required, opened a new log
I20260925 16:32:59.839730 2642 ts_tablet_manager.cc:1397] T 4ff1cb81a17041e18460ada65b07f1cc P a3d3aa90f8e74bb2ae128513c9f0523a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 16:32:59.840191 2642 raft_consensus.cc:348] T 4ff1cb81a17041e18460ada65b07f1cc P a3d3aa90f8e74bb2ae128513c9f0523a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435b9fb0fe93430face8f32e4de081b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43805 } } peers { permanent_uuid: "97509c025dbe428fabba21071efa72ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37461 } } peers { permanent_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43817 } }
I20260925 16:32:59.840297 2642 raft_consensus.cc:374] T 4ff1cb81a17041e18460ada65b07f1cc P a3d3aa90f8e74bb2ae128513c9f0523a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:32:59.840356 2642 raft_consensus.cc:729] T 4ff1cb81a17041e18460ada65b07f1cc P a3d3aa90f8e74bb2ae128513c9f0523a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3d3aa90f8e74bb2ae128513c9f0523a, State: Initialized, Role: FOLLOWER
I20260925 16:32:59.840476 2642 consensus_queue.cc:260] T 4ff1cb81a17041e18460ada65b07f1cc P a3d3aa90f8e74bb2ae128513c9f0523a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435b9fb0fe93430face8f32e4de081b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43805 } } peers { permanent_uuid: "97509c025dbe428fabba21071efa72ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37461 } } peers { permanent_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43817 } }
I20260925 16:32:59.840737 2642 ts_tablet_manager.cc:1428] T 4ff1cb81a17041e18460ada65b07f1cc P a3d3aa90f8e74bb2ae128513c9f0523a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 16:32:59.840695 2650 tablet_replica.cc:439] T 4ff1cb81a17041e18460ada65b07f1cc P a3d3aa90f8e74bb2ae128513c9f0523a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435b9fb0fe93430face8f32e4de081b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43805 } } peers { permanent_uuid: "97509c025dbe428fabba21071efa72ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37461 } } peers { permanent_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43817 } } }
I20260925 16:32:59.840808 2650 tablet_replica.cc:442] T 4ff1cb81a17041e18460ada65b07f1cc P a3d3aa90f8e74bb2ae128513c9f0523a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 16:33:00.017679 2649 raft_consensus.cc:482] T 4ff1cb81a17041e18460ada65b07f1cc P 435b9fb0fe93430face8f32e4de081b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:33:00.017813 2649 raft_consensus.cc:504] T 4ff1cb81a17041e18460ada65b07f1cc P 435b9fb0fe93430face8f32e4de081b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435b9fb0fe93430face8f32e4de081b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43805 } } peers { permanent_uuid: "97509c025dbe428fabba21071efa72ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37461 } } peers { permanent_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43817 } }
I20260925 16:33:00.019001 2649 leader_election.cc:288] T 4ff1cb81a17041e18460ada65b07f1cc P 435b9fb0fe93430face8f32e4de081b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97509c025dbe428fabba21071efa72ce (127.0.98.2:37461), a3d3aa90f8e74bb2ae128513c9f0523a (127.0.98.3:43817)
I20260925 16:33:00.024232 2663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:00.019226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37461 (local address 127.0.98.1:43679)
0925 16:33:00.019578 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:00.019594 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:00.019704 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:00.019956 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:00.019961 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:00.019976 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:33:00.020168 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:00.020176 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:00.021294 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:00.021297 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:00.022895 (+ 1598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:00.022906 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:00.023774 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:00.023778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:00.023868 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:00.023891 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 16:33:00.023979 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 16:33:00.024039 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":176,"thread_start_us":130,"threads_started":1}
I20260925 16:33:00.024241 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:00.019621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43679 (local address 127.0.98.2:37461)
0925 16:33:00.019732 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:00.019736 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:00.019751 (+ 15us) server_negotiation.cc:407] Connection header received
0925 16:33:00.019806 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:00.019809 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:00.019852 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:00.019926 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:33:00.020311 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:00.021144 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:00.023029 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:00.023623 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:00.023737 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:00.023801 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:00.024002 (+ 201us) server_negotiation.cc:299] Negotiation successful
0925 16:33:00.024076 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 16:33:00.024684 2664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:00.019232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43817 (local address 127.0.98.1:49747)
0925 16:33:00.019791 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:00.019804 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:00.019917 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:00.020258 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:00.020261 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:00.020276 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:33:00.020369 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:00.020374 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:00.021337 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:00.021339 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:00.022895 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:00.022906 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:00.024236 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:00.024242 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:00.024361 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:00.024399 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 16:33:00.024491 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 16:33:00.024565 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"mutex_wait_us":114,"thread_start_us":138,"threads_started":1}
I20260925 16:33:00.024771 2640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:00.019380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49747 (local address 127.0.98.3:43817)
0925 16:33:00.019593 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:00.019599 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:00.019886 (+ 287us) server_negotiation.cc:407] Connection header received
0925 16:33:00.020045 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:00.020049 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:00.020097 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:00.020163 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:33:00.020489 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:00.021200 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:00.023511 (+ 2311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:00.024086 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:00.024207 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:00.024270 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:00.024535 (+ 265us) server_negotiation.cc:299] Negotiation successful
0925 16:33:00.024626 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
I20260925 16:33:00.025053 2433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ff1cb81a17041e18460ada65b07f1cc" candidate_uuid: "435b9fb0fe93430face8f32e4de081b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97509c025dbe428fabba21071efa72ce" is_pre_election: true
I20260925 16:33:00.025214 2433 raft_consensus.cc:2454] T 4ff1cb81a17041e18460ada65b07f1cc P 97509c025dbe428fabba21071efa72ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 435b9fb0fe93430face8f32e4de081b4 in term 0.
I20260925 16:33:00.025365 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ff1cb81a17041e18460ada65b07f1cc" candidate_uuid: "435b9fb0fe93430face8f32e4de081b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" is_pre_election: true
I20260925 16:33:00.025564 2227 leader_election.cc:302] T 4ff1cb81a17041e18460ada65b07f1cc P 435b9fb0fe93430face8f32e4de081b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 435b9fb0fe93430face8f32e4de081b4, 97509c025dbe428fabba21071efa72ce; no voters:
I20260925 16:33:00.025629 2572 raft_consensus.cc:2454] T 4ff1cb81a17041e18460ada65b07f1cc P a3d3aa90f8e74bb2ae128513c9f0523a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 435b9fb0fe93430face8f32e4de081b4 in term 0.
I20260925 16:33:00.025857 2649 raft_consensus.cc:2781] T 4ff1cb81a17041e18460ada65b07f1cc P 435b9fb0fe93430face8f32e4de081b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:33:00.025947 2649 raft_consensus.cc:482] T 4ff1cb81a17041e18460ada65b07f1cc P 435b9fb0fe93430face8f32e4de081b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:33:00.025992 2649 raft_consensus.cc:3037] T 4ff1cb81a17041e18460ada65b07f1cc P 435b9fb0fe93430face8f32e4de081b4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:33:00.027056 2649 raft_consensus.cc:504] T 4ff1cb81a17041e18460ada65b07f1cc P 435b9fb0fe93430face8f32e4de081b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435b9fb0fe93430face8f32e4de081b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43805 } } peers { permanent_uuid: "97509c025dbe428fabba21071efa72ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37461 } } peers { permanent_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43817 } }
I20260925 16:33:00.027456 2649 leader_election.cc:288] T 4ff1cb81a17041e18460ada65b07f1cc P 435b9fb0fe93430face8f32e4de081b4 [CANDIDATE]: Term 1 election: Requested vote from peers 97509c025dbe428fabba21071efa72ce (127.0.98.2:37461), a3d3aa90f8e74bb2ae128513c9f0523a (127.0.98.3:43817)
I20260925 16:33:00.027768 2433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ff1cb81a17041e18460ada65b07f1cc" candidate_uuid: "435b9fb0fe93430face8f32e4de081b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97509c025dbe428fabba21071efa72ce"
I20260925 16:33:00.027884 2433 raft_consensus.cc:3037] T 4ff1cb81a17041e18460ada65b07f1cc P 97509c025dbe428fabba21071efa72ce [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:33:00.027838 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ff1cb81a17041e18460ada65b07f1cc" candidate_uuid: "435b9fb0fe93430face8f32e4de081b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a"
I20260925 16:33:00.027987 2572 raft_consensus.cc:3037] T 4ff1cb81a17041e18460ada65b07f1cc P a3d3aa90f8e74bb2ae128513c9f0523a [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:33:00.028837 2433 raft_consensus.cc:2454] T 4ff1cb81a17041e18460ada65b07f1cc P 97509c025dbe428fabba21071efa72ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 435b9fb0fe93430face8f32e4de081b4 in term 1.
I20260925 16:33:00.029012 2572 raft_consensus.cc:2454] T 4ff1cb81a17041e18460ada65b07f1cc P a3d3aa90f8e74bb2ae128513c9f0523a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 435b9fb0fe93430face8f32e4de081b4 in term 1.
I20260925 16:33:00.029219 2227 leader_election.cc:302] T 4ff1cb81a17041e18460ada65b07f1cc P 435b9fb0fe93430face8f32e4de081b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 435b9fb0fe93430face8f32e4de081b4, 97509c025dbe428fabba21071efa72ce; no voters:
I20260925 16:33:00.029403 2649 raft_consensus.cc:2781] T 4ff1cb81a17041e18460ada65b07f1cc P 435b9fb0fe93430face8f32e4de081b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:33:00.029647 2649 raft_consensus.cc:686] T 4ff1cb81a17041e18460ada65b07f1cc P 435b9fb0fe93430face8f32e4de081b4 [term 1 LEADER]: Becoming Leader. State: Replica: 435b9fb0fe93430face8f32e4de081b4, State: Running, Role: LEADER
I20260925 16:33:00.029982 2649 consensus_queue.cc:237] T 4ff1cb81a17041e18460ada65b07f1cc P 435b9fb0fe93430face8f32e4de081b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435b9fb0fe93430face8f32e4de081b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43805 } } peers { permanent_uuid: "97509c025dbe428fabba21071efa72ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37461 } } peers { permanent_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43817 } }
I20260925 16:33:00.032065 2665 tablet_replica.cc:439] T 4ff1cb81a17041e18460ada65b07f1cc P 435b9fb0fe93430face8f32e4de081b4: TxnStatusTablet state changed. Reason: New leader 435b9fb0fe93430face8f32e4de081b4. Latest consensus state: current_term: 1 leader_uuid: "435b9fb0fe93430face8f32e4de081b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435b9fb0fe93430face8f32e4de081b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43805 } } peers { permanent_uuid: "97509c025dbe428fabba21071efa72ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37461 } } peers { permanent_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43817 } } }
I20260925 16:33:00.032181 2665 tablet_replica.cc:442] T 4ff1cb81a17041e18460ada65b07f1cc P 435b9fb0fe93430face8f32e4de081b4: This TxnStatusTablet replica's current role is: LEADER
I20260925 16:33:00.032658 2667 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 16:33:00.032855 2667 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 16:33:00.033051 2135 catalog_manager.cc:5495] T 4ff1cb81a17041e18460ada65b07f1cc P 435b9fb0fe93430face8f32e4de081b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 435b9fb0fe93430face8f32e4de081b4 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "435b9fb0fe93430face8f32e4de081b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435b9fb0fe93430face8f32e4de081b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97509c025dbe428fabba21071efa72ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43817 } health_report { overall_health: UNKNOWN } } }
I20260925 16:33:00.077416 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:00.073253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58404 (local address 127.0.98.1:43805)
0925 16:33:00.073406 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:00.073411 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:00.073470 (+ 59us) server_negotiation.cc:407] Connection header received
0925 16:33:00.073705 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:00.073710 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:00.073787 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:00.073884 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:33:00.074179 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:00.074775 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:00.076032 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:00.076249 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:00.076327 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:00.076641 (+ 314us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:33:00.077063 (+ 422us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:33:00.077175 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:00.077256 (+ 81us) server_negotiation.cc:299] Negotiation successful
0925 16:33:00.077301 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 16:33:00.077481 2659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:00.073141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43805 (local address 127.0.0.1:58404)
0925 16:33:00.073407 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:00.073424 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:00.073541 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:00.073906 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:00.073909 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:00.073916 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 16:33:00.074022 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:00.074030 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:00.074927 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:00.074931 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:00.075918 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:00.075929 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:00.076352 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:00.076355 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:00.076448 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:00.076501 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 16:33:00.077157 (+ 656us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 16:33:00.077171 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 16:33:00.077247 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 16:33:00.077338 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161}
I20260925 16:33:00.082841 2572 raft_consensus.cc:1261] T 4ff1cb81a17041e18460ada65b07f1cc P a3d3aa90f8e74bb2ae128513c9f0523a [term 1 FOLLOWER]: Refusing update from remote peer 435b9fb0fe93430face8f32e4de081b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:33:00.082944 2433 raft_consensus.cc:1261] T 4ff1cb81a17041e18460ada65b07f1cc P 97509c025dbe428fabba21071efa72ce [term 1 FOLLOWER]: Refusing update from remote peer 435b9fb0fe93430face8f32e4de081b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:33:00.083559 2665 consensus_queue.cc:1035] T 4ff1cb81a17041e18460ada65b07f1cc P 435b9fb0fe93430face8f32e4de081b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:33:00.083743 2649 consensus_queue.cc:1035] T 4ff1cb81a17041e18460ada65b07f1cc P 435b9fb0fe93430face8f32e4de081b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97509c025dbe428fabba21071efa72ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:33:00.086678 2650 tablet_replica.cc:439] T 4ff1cb81a17041e18460ada65b07f1cc P a3d3aa90f8e74bb2ae128513c9f0523a: TxnStatusTablet state changed. Reason: New leader 435b9fb0fe93430face8f32e4de081b4. Latest consensus state: current_term: 1 leader_uuid: "435b9fb0fe93430face8f32e4de081b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435b9fb0fe93430face8f32e4de081b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43805 } } peers { permanent_uuid: "97509c025dbe428fabba21071efa72ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37461 } } peers { permanent_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43817 } } }
I20260925 16:33:00.086792 2650 tablet_replica.cc:442] T 4ff1cb81a17041e18460ada65b07f1cc P a3d3aa90f8e74bb2ae128513c9f0523a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 16:33:00.087571 2648 tablet_replica.cc:439] T 4ff1cb81a17041e18460ada65b07f1cc P 97509c025dbe428fabba21071efa72ce: TxnStatusTablet state changed. Reason: New leader 435b9fb0fe93430face8f32e4de081b4. Latest consensus state: current_term: 1 leader_uuid: "435b9fb0fe93430face8f32e4de081b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435b9fb0fe93430face8f32e4de081b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43805 } } peers { permanent_uuid: "97509c025dbe428fabba21071efa72ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37461 } } peers { permanent_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43817 } } }
I20260925 16:33:00.087699 2648 tablet_replica.cc:442] T 4ff1cb81a17041e18460ada65b07f1cc P 97509c025dbe428fabba21071efa72ce: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 16:33:00.091149 2649 tablet_replica.cc:439] T 4ff1cb81a17041e18460ada65b07f1cc P 435b9fb0fe93430face8f32e4de081b4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "435b9fb0fe93430face8f32e4de081b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435b9fb0fe93430face8f32e4de081b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43805 } } peers { permanent_uuid: "97509c025dbe428fabba21071efa72ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37461 } } peers { permanent_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43817 } } }
I20260925 16:33:00.091274 2649 tablet_replica.cc:442] T 4ff1cb81a17041e18460ada65b07f1cc P 435b9fb0fe93430face8f32e4de081b4: This TxnStatusTablet replica's current role is: LEADER
I20260925 16:33:00.091948 2650 tablet_replica.cc:439] T 4ff1cb81a17041e18460ada65b07f1cc P a3d3aa90f8e74bb2ae128513c9f0523a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "435b9fb0fe93430face8f32e4de081b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435b9fb0fe93430face8f32e4de081b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43805 } } peers { permanent_uuid: "97509c025dbe428fabba21071efa72ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37461 } } peers { permanent_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43817 } } }
I20260925 16:33:00.092062 2650 tablet_replica.cc:442] T 4ff1cb81a17041e18460ada65b07f1cc P a3d3aa90f8e74bb2ae128513c9f0523a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 16:33:00.093719 2672 mvcc.cc:205] Tried to move back new op lower bound from 7333289902412005376 to 7333289902205521920. Current Snapshot: MvccSnapshot[applied={T|T < 7333289902412005376}]
I20260925 16:33:00.094677 2649 tablet_replica.cc:439] T 4ff1cb81a17041e18460ada65b07f1cc P 435b9fb0fe93430face8f32e4de081b4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "435b9fb0fe93430face8f32e4de081b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435b9fb0fe93430face8f32e4de081b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43805 } } peers { permanent_uuid: "97509c025dbe428fabba21071efa72ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37461 } } peers { permanent_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43817 } } }
I20260925 16:33:00.094782 2649 tablet_replica.cc:442] T 4ff1cb81a17041e18460ada65b07f1cc P 435b9fb0fe93430face8f32e4de081b4: This TxnStatusTablet replica's current role is: LEADER
I20260925 16:33:00.095824 2671 mvcc.cc:205] Tried to move back new op lower bound from 7333289902412005376 to 7333289902205521920. Current Snapshot: MvccSnapshot[applied={T|T < 7333289902412005376}]
I20260925 16:33:00.095999 2648 tablet_replica.cc:439] T 4ff1cb81a17041e18460ada65b07f1cc P 97509c025dbe428fabba21071efa72ce: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "435b9fb0fe93430face8f32e4de081b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435b9fb0fe93430face8f32e4de081b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43805 } } peers { permanent_uuid: "97509c025dbe428fabba21071efa72ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37461 } } peers { permanent_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43817 } } }
I20260925 16:33:00.096108 2648 tablet_replica.cc:442] T 4ff1cb81a17041e18460ada65b07f1cc P 97509c025dbe428fabba21071efa72ce: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 16:33:00.102633 2640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:00.098235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41494 (local address 127.0.98.3:43817)
0925 16:33:00.098346 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:00.098350 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:00.098503 (+ 153us) server_negotiation.cc:407] Connection header received
0925 16:33:00.098638 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:00.098641 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:00.098699 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:00.098808 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:33:00.099051 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:00.099952 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:00.100899 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:00.101312 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:00.101418 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:00.101720 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:33:00.102223 (+ 503us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:33:00.102336 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:00.102413 (+ 77us) server_negotiation.cc:299] Negotiation successful
0925 16:33:00.102469 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 16:33:00.113706 2683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:00.110003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43805 (local address 127.0.98.3:41721)
0925 16:33:00.110382 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:00.110394 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:00.110482 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:00.110732 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:00.110734 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:00.110741 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 16:33:00.110839 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:00.110843 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:00.111840 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:00.111842 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:00.112455 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:00.112462 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:00.113012 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:00.113015 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:00.113081 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:00.113148 (+ 67us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 16:33:00.113460 (+ 312us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 16:33:00.113464 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 16:33:00.113514 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 16:33:00.113575 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":135,"threads_started":1}
I20260925 16:33:00.113736 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:00.110113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41721 (local address 127.0.98.1:43805)
0925 16:33:00.110240 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:00.110243 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:00.110479 (+ 236us) server_negotiation.cc:407] Connection header received
0925 16:33:00.110559 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:00.110562 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:00.110639 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:00.110718 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:33:00.110947 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:00.111739 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:00.112565 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:00.112910 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:00.113008 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:00.113297 (+ 289us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:33:00.113378 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:33:00.113449 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:00.113550 (+ 101us) server_negotiation.cc:299] Negotiation successful
0925 16:33:00.113607 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 16:33:00.121119 2433 raft_consensus.cc:1261] T 0c4e5abf31d743958b58b4089a56ec9f P 97509c025dbe428fabba21071efa72ce [term 1 FOLLOWER]: Refusing update from remote peer a3d3aa90f8e74bb2ae128513c9f0523a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:33:00.121184 2293 raft_consensus.cc:1261] T 0c4e5abf31d743958b58b4089a56ec9f P 435b9fb0fe93430face8f32e4de081b4 [term 1 FOLLOWER]: Refusing update from remote peer a3d3aa90f8e74bb2ae128513c9f0523a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:33:00.121673 2650 consensus_queue.cc:1035] T 0c4e5abf31d743958b58b4089a56ec9f P a3d3aa90f8e74bb2ae128513c9f0523a [LEADER]: Connected to new peer: Peer: permanent_uuid: "97509c025dbe428fabba21071efa72ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:33:00.121873 2652 consensus_queue.cc:1035] T 0c4e5abf31d743958b58b4089a56ec9f P a3d3aa90f8e74bb2ae128513c9f0523a [LEADER]: Connected to new peer: Peer: permanent_uuid: "435b9fb0fe93430face8f32e4de081b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:33:00.127838 2670 mvcc.cc:205] Tried to move back new op lower bound from 7333289902571323392 to 7333289901167099904. Current Snapshot: MvccSnapshot[applied={T|T < 7333289902571323392}]
I20260925 16:33:00.147226 2640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:00.142925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:32901 (local address 127.0.98.3:43817)
0925 16:33:00.143212 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:00.143219 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:00.143237 (+ 18us) server_negotiation.cc:407] Connection header received
0925 16:33:00.143405 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:00.143408 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:00.143463 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:00.143544 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:33:00.143799 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:00.144652 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:00.145914 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:00.146282 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:00.146390 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:00.146812 (+ 422us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:33:00.146890 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:33:00.146959 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:00.147071 (+ 112us) server_negotiation.cc:299] Negotiation successful
0925 16:33:00.147116 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214}
I20260925 16:33:00.147378 2688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:00.142796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43817 (local address 127.0.98.1:32901)
0925 16:33:00.143160 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:00.143179 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:00.143282 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:00.143560 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:00.143562 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:00.143568 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 16:33:00.143699 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:00.143704 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:00.144819 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:00.144823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:00.145763 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:00.145774 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:00.146492 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:00.146498 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:00.146625 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:00.146677 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 16:33:00.146979 (+ 302us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 16:33:00.146986 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 16:33:00.147060 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 16:33:00.147145 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":130,"threads_started":1}
I20260925 16:33:00.176862 2136 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:33:00.177055 2136 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:33:00.179916 2136 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2325bb98933b48e880649d64cbdac636: Sending DeleteTablet for 3 replicas of tablet 0c4e5abf31d743958b58b4089a56ec9f
I20260925 16:33:00.180631 2273 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c4e5abf31d743958b58b4089a56ec9f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:33:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58388
I20260925 16:33:00.180974 2413 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c4e5abf31d743958b58b4089a56ec9f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:33:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37916
I20260925 16:33:00.181311 2691 tablet_replica.cc:330] T 0c4e5abf31d743958b58b4089a56ec9f P 435b9fb0fe93430face8f32e4de081b4: stopping tablet replica
I20260925 16:33:00.181523 2553 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c4e5abf31d743958b58b4089a56ec9f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:33:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41490
I20260925 16:33:00.182214 2691 raft_consensus.cc:2229] T 0c4e5abf31d743958b58b4089a56ec9f P 435b9fb0fe93430face8f32e4de081b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:33:00.182463 2691 raft_consensus.cc:2258] T 0c4e5abf31d743958b58b4089a56ec9f P 435b9fb0fe93430face8f32e4de081b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:33:00.182547 2136 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55664:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 16:33:00.182997 2136 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:33:00.183936 2692 tablet_replica.cc:330] T 0c4e5abf31d743958b58b4089a56ec9f P 97509c025dbe428fabba21071efa72ce: stopping tablet replica
I20260925 16:33:00.184199 2692 raft_consensus.cc:2229] T 0c4e5abf31d743958b58b4089a56ec9f P 97509c025dbe428fabba21071efa72ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:33:00.184374 2692 raft_consensus.cc:2258] T 0c4e5abf31d743958b58b4089a56ec9f P 97509c025dbe428fabba21071efa72ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:33:00.186401 2691 ts_tablet_manager.cc:1905] T 0c4e5abf31d743958b58b4089a56ec9f P 435b9fb0fe93430face8f32e4de081b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:33:00.188777 2273 tablet_service.cc:1467] Processing CreateTablet for tablet 47c334f57049409294e227c6e26311e8 (DEFAULT_TABLE table=test-table [id=39970adee75643dca9b2a4909d0a29cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:33:00.189036 2273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47c334f57049409294e227c6e26311e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:33:00.190049 2691 ts_tablet_manager.cc:1918] T 0c4e5abf31d743958b58b4089a56ec9f P 435b9fb0fe93430face8f32e4de081b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 16:33:00.190244 2691 log.cc:1196] T 0c4e5abf31d743958b58b4089a56ec9f P 435b9fb0fe93430face8f32e4de081b4: Deleting WAL directory at /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/ts-0/wal/wals/0c4e5abf31d743958b58b4089a56ec9f
I20260925 16:33:00.190698 2691 ts_tablet_manager.cc:1939] T 0c4e5abf31d743958b58b4089a56ec9f P 435b9fb0fe93430face8f32e4de081b4: Deleting consensus metadata
I20260925 16:33:00.190868 2553 tablet_service.cc:1467] Processing CreateTablet for tablet 47c334f57049409294e227c6e26311e8 (DEFAULT_TABLE table=test-table [id=39970adee75643dca9b2a4909d0a29cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:33:00.191125 2553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47c334f57049409294e227c6e26311e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:33:00.193578 2642 tablet_bootstrap.cc:492] T 47c334f57049409294e227c6e26311e8 P a3d3aa90f8e74bb2ae128513c9f0523a: Bootstrap starting.
I20260925 16:33:00.193950 2121 catalog_manager.cc:4841] TS 435b9fb0fe93430face8f32e4de081b4 (127.0.98.1:43805): tablet 0c4e5abf31d743958b58b4089a56ec9f (table test-table [id=a7eab0096a42478c9af60eb344c780a0]) successfully deleted
I20260925 16:33:00.194770 2642 tablet_bootstrap.cc:654] T 47c334f57049409294e227c6e26311e8 P a3d3aa90f8e74bb2ae128513c9f0523a: Neither blocks nor log segments found. Creating new log.
I20260925 16:33:00.195067 2413 tablet_service.cc:1467] Processing CreateTablet for tablet 47c334f57049409294e227c6e26311e8 (DEFAULT_TABLE table=test-table [id=39970adee75643dca9b2a4909d0a29cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:33:00.195318 2413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47c334f57049409294e227c6e26311e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:33:00.196142 2642 tablet_bootstrap.cc:492] T 47c334f57049409294e227c6e26311e8 P a3d3aa90f8e74bb2ae128513c9f0523a: No bootstrap required, opened a new log
I20260925 16:33:00.196228 2642 ts_tablet_manager.cc:1397] T 47c334f57049409294e227c6e26311e8 P a3d3aa90f8e74bb2ae128513c9f0523a: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260925 16:33:00.196673 2642 raft_consensus.cc:348] T 47c334f57049409294e227c6e26311e8 P a3d3aa90f8e74bb2ae128513c9f0523a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435b9fb0fe93430face8f32e4de081b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43805 } } peers { permanent_uuid: "97509c025dbe428fabba21071efa72ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37461 } } peers { permanent_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43817 } }
I20260925 16:33:00.196784 2642 raft_consensus.cc:374] T 47c334f57049409294e227c6e26311e8 P a3d3aa90f8e74bb2ae128513c9f0523a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:33:00.196815 2642 raft_consensus.cc:729] T 47c334f57049409294e227c6e26311e8 P a3d3aa90f8e74bb2ae128513c9f0523a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3d3aa90f8e74bb2ae128513c9f0523a, State: Initialized, Role: FOLLOWER
I20260925 16:33:00.196938 2642 consensus_queue.cc:260] T 47c334f57049409294e227c6e26311e8 P a3d3aa90f8e74bb2ae128513c9f0523a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435b9fb0fe93430face8f32e4de081b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43805 } } peers { permanent_uuid: "97509c025dbe428fabba21071efa72ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37461 } } peers { permanent_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43817 } }
I20260925 16:33:00.197222 2642 ts_tablet_manager.cc:1428] T 47c334f57049409294e227c6e26311e8 P a3d3aa90f8e74bb2ae128513c9f0523a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260925 16:33:00.197718 2644 tablet_bootstrap.cc:492] T 47c334f57049409294e227c6e26311e8 P 97509c025dbe428fabba21071efa72ce: Bootstrap starting.
I20260925 16:33:00.198908 2693 tablet_replica.cc:330] T 0c4e5abf31d743958b58b4089a56ec9f P a3d3aa90f8e74bb2ae128513c9f0523a: stopping tablet replica
I20260925 16:33:00.199076 2644 tablet_bootstrap.cc:654] T 47c334f57049409294e227c6e26311e8 P 97509c025dbe428fabba21071efa72ce: Neither blocks nor log segments found. Creating new log.
I20260925 16:33:00.199311 2693 raft_consensus.cc:2229] T 0c4e5abf31d743958b58b4089a56ec9f P a3d3aa90f8e74bb2ae128513c9f0523a [term 1 LEADER]: Raft consensus shutting down.
I20260925 16:33:00.199728 2693 raft_consensus.cc:2258] T 0c4e5abf31d743958b58b4089a56ec9f P a3d3aa90f8e74bb2ae128513c9f0523a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:33:00.200169 2644 tablet_bootstrap.cc:492] T 47c334f57049409294e227c6e26311e8 P 97509c025dbe428fabba21071efa72ce: No bootstrap required, opened a new log
I20260925 16:33:00.200246 2644 ts_tablet_manager.cc:1397] T 47c334f57049409294e227c6e26311e8 P 97509c025dbe428fabba21071efa72ce: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260925 16:33:00.200732 2644 raft_consensus.cc:348] T 47c334f57049409294e227c6e26311e8 P 97509c025dbe428fabba21071efa72ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435b9fb0fe93430face8f32e4de081b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43805 } } peers { permanent_uuid: "97509c025dbe428fabba21071efa72ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37461 } } peers { permanent_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43817 } }
I20260925 16:33:00.200873 2644 raft_consensus.cc:374] T 47c334f57049409294e227c6e26311e8 P 97509c025dbe428fabba21071efa72ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:33:00.200914 2644 raft_consensus.cc:729] T 47c334f57049409294e227c6e26311e8 P 97509c025dbe428fabba21071efa72ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97509c025dbe428fabba21071efa72ce, State: Initialized, Role: FOLLOWER
I20260925 16:33:00.201057 2644 consensus_queue.cc:260] T 47c334f57049409294e227c6e26311e8 P 97509c025dbe428fabba21071efa72ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435b9fb0fe93430face8f32e4de081b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43805 } } peers { permanent_uuid: "97509c025dbe428fabba21071efa72ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37461 } } peers { permanent_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43817 } }
I20260925 16:33:00.201280 2693 ts_tablet_manager.cc:1905] T 0c4e5abf31d743958b58b4089a56ec9f P a3d3aa90f8e74bb2ae128513c9f0523a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:33:00.201299 2644 ts_tablet_manager.cc:1428] T 47c334f57049409294e227c6e26311e8 P 97509c025dbe428fabba21071efa72ce: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260925 16:33:00.201634 2643 tablet_bootstrap.cc:492] T 47c334f57049409294e227c6e26311e8 P 435b9fb0fe93430face8f32e4de081b4: Bootstrap starting.
I20260925 16:33:00.202664 2643 tablet_bootstrap.cc:654] T 47c334f57049409294e227c6e26311e8 P 435b9fb0fe93430face8f32e4de081b4: Neither blocks nor log segments found. Creating new log.
I20260925 16:33:00.204144 2692 ts_tablet_manager.cc:1905] T 0c4e5abf31d743958b58b4089a56ec9f P 97509c025dbe428fabba21071efa72ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:33:00.204731 2693 ts_tablet_manager.cc:1918] T 0c4e5abf31d743958b58b4089a56ec9f P a3d3aa90f8e74bb2ae128513c9f0523a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 16:33:00.204844 2693 log.cc:1196] T 0c4e5abf31d743958b58b4089a56ec9f P a3d3aa90f8e74bb2ae128513c9f0523a: Deleting WAL directory at /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/ts-2/wal/wals/0c4e5abf31d743958b58b4089a56ec9f
I20260925 16:33:00.205192 2693 ts_tablet_manager.cc:1939] T 0c4e5abf31d743958b58b4089a56ec9f P a3d3aa90f8e74bb2ae128513c9f0523a: Deleting consensus metadata
I20260925 16:33:00.205598 2643 tablet_bootstrap.cc:492] T 47c334f57049409294e227c6e26311e8 P 435b9fb0fe93430face8f32e4de081b4: No bootstrap required, opened a new log
I20260925 16:33:00.205691 2643 ts_tablet_manager.cc:1397] T 47c334f57049409294e227c6e26311e8 P 435b9fb0fe93430face8f32e4de081b4: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 16:33:00.206077 2643 raft_consensus.cc:348] T 47c334f57049409294e227c6e26311e8 P 435b9fb0fe93430face8f32e4de081b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435b9fb0fe93430face8f32e4de081b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43805 } } peers { permanent_uuid: "97509c025dbe428fabba21071efa72ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37461 } } peers { permanent_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43817 } }
I20260925 16:33:00.206194 2643 raft_consensus.cc:374] T 47c334f57049409294e227c6e26311e8 P 435b9fb0fe93430face8f32e4de081b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:33:00.206226 2643 raft_consensus.cc:729] T 47c334f57049409294e227c6e26311e8 P 435b9fb0fe93430face8f32e4de081b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 435b9fb0fe93430face8f32e4de081b4, State: Initialized, Role: FOLLOWER
I20260925 16:33:00.206342 2643 consensus_queue.cc:260] T 47c334f57049409294e227c6e26311e8 P 435b9fb0fe93430face8f32e4de081b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435b9fb0fe93430face8f32e4de081b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43805 } } peers { permanent_uuid: "97509c025dbe428fabba21071efa72ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37461 } } peers { permanent_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43817 } }
I20260925 16:33:00.206461 2122 catalog_manager.cc:4841] TS a3d3aa90f8e74bb2ae128513c9f0523a (127.0.98.3:43817): tablet 0c4e5abf31d743958b58b4089a56ec9f (table test-table [id=a7eab0096a42478c9af60eb344c780a0]) successfully deleted
I20260925 16:33:00.206627 2643 ts_tablet_manager.cc:1428] T 47c334f57049409294e227c6e26311e8 P 435b9fb0fe93430face8f32e4de081b4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 16:33:00.208117 2692 ts_tablet_manager.cc:1918] T 0c4e5abf31d743958b58b4089a56ec9f P 97509c025dbe428fabba21071efa72ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 16:33:00.208262 2692 log.cc:1196] T 0c4e5abf31d743958b58b4089a56ec9f P 97509c025dbe428fabba21071efa72ce: Deleting WAL directory at /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/ts-1/wal/wals/0c4e5abf31d743958b58b4089a56ec9f
I20260925 16:33:00.208652 2692 ts_tablet_manager.cc:1939] T 0c4e5abf31d743958b58b4089a56ec9f P 97509c025dbe428fabba21071efa72ce: Deleting consensus metadata
I20260925 16:33:00.209915 2123 catalog_manager.cc:4841] TS 97509c025dbe428fabba21071efa72ce (127.0.98.2:37461): tablet 0c4e5abf31d743958b58b4089a56ec9f (table test-table [id=a7eab0096a42478c9af60eb344c780a0]) successfully deleted
I20260925 16:33:00.400197 2648 raft_consensus.cc:482] T 47c334f57049409294e227c6e26311e8 P 97509c025dbe428fabba21071efa72ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:33:00.400352 2648 raft_consensus.cc:504] T 47c334f57049409294e227c6e26311e8 P 97509c025dbe428fabba21071efa72ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435b9fb0fe93430face8f32e4de081b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43805 } } peers { permanent_uuid: "97509c025dbe428fabba21071efa72ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37461 } } peers { permanent_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43817 } }
I20260925 16:33:00.401427 2648 leader_election.cc:288] T 47c334f57049409294e227c6e26311e8 P 97509c025dbe428fabba21071efa72ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 435b9fb0fe93430face8f32e4de081b4 (127.0.98.1:43805), a3d3aa90f8e74bb2ae128513c9f0523a (127.0.98.3:43817)
I20260925 16:33:00.406412 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:00.401798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44735 (local address 127.0.98.1:43805)
0925 16:33:00.401986 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:00.401994 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:00.402148 (+ 154us) server_negotiation.cc:407] Connection header received
0925 16:33:00.402327 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:00.402333 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:00.402388 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:00.402446 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:33:00.402858 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:00.403389 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:00.404879 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:00.405565 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:00.405670 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:00.405760 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:00.406193 (+ 433us) server_negotiation.cc:299] Negotiation successful
0925 16:33:00.406272 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 16:33:00.406872 2698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:00.401648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43805 (local address 127.0.98.2:44735)
0925 16:33:00.402009 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:00.402023 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:00.402200 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:00.402490 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:00.402496 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:00.402515 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:33:00.402710 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:00.402718 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:00.403521 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:00.403525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:00.404735 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:00.404748 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:00.405902 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:00.405906 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:00.406016 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:00.406041 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 16:33:00.406561 (+ 520us) client_negotiation.cc:240] Negotiation successful
0925 16:33:00.406669 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"mutex_wait_us":5,"spinlock_wait_cycles":3840,"thread_start_us":108,"threads_started":1}
I20260925 16:33:00.407021 2640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:00.401794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48629 (local address 127.0.98.3:43817)
0925 16:33:00.401934 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:00.401940 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:00.402206 (+ 266us) server_negotiation.cc:407] Connection header received
0925 16:33:00.402360 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:00.402363 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:00.402408 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:00.402571 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:33:00.403568 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:00.404303 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:00.405767 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:00.406366 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:00.406442 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:00.406501 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:00.406832 (+ 331us) server_negotiation.cc:299] Negotiation successful
0925 16:33:00.406906 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 16:33:00.407656 2697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:00.401648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43817 (local address 127.0.98.2:48629)
0925 16:33:00.402079 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:00.402096 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:00.402216 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:00.402533 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:00.402536 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:00.402569 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:33:00.402726 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:00.402732 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:00.404437 (+ 1705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:00.404440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:00.405612 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:00.405622 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:00.406614 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:00.406617 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:00.406702 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:00.406723 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 16:33:00.407202 (+ 479us) client_negotiation.cc:240] Negotiation successful
0925 16:33:00.407271 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"mutex_wait_us":52,"thread_start_us":71,"threads_started":1}
I20260925 16:33:00.408142 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c334f57049409294e227c6e26311e8" candidate_uuid: "97509c025dbe428fabba21071efa72ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" is_pre_election: true
I20260925 16:33:00.408300 2572 raft_consensus.cc:2454] T 47c334f57049409294e227c6e26311e8 P a3d3aa90f8e74bb2ae128513c9f0523a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97509c025dbe428fabba21071efa72ce in term 0.
I20260925 16:33:00.408681 2366 leader_election.cc:302] T 47c334f57049409294e227c6e26311e8 P 97509c025dbe428fabba21071efa72ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97509c025dbe428fabba21071efa72ce, a3d3aa90f8e74bb2ae128513c9f0523a; no voters:
I20260925 16:33:00.408975 2648 raft_consensus.cc:2781] T 47c334f57049409294e227c6e26311e8 P 97509c025dbe428fabba21071efa72ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:33:00.409065 2648 raft_consensus.cc:482] T 47c334f57049409294e227c6e26311e8 P 97509c025dbe428fabba21071efa72ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:33:00.409116 2648 raft_consensus.cc:3037] T 47c334f57049409294e227c6e26311e8 P 97509c025dbe428fabba21071efa72ce [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:33:00.409377 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c334f57049409294e227c6e26311e8" candidate_uuid: "97509c025dbe428fabba21071efa72ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "435b9fb0fe93430face8f32e4de081b4" is_pre_election: true
I20260925 16:33:00.409528 2293 raft_consensus.cc:2454] T 47c334f57049409294e227c6e26311e8 P 435b9fb0fe93430face8f32e4de081b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97509c025dbe428fabba21071efa72ce in term 0.
I20260925 16:33:00.410734 2648 raft_consensus.cc:504] T 47c334f57049409294e227c6e26311e8 P 97509c025dbe428fabba21071efa72ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435b9fb0fe93430face8f32e4de081b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43805 } } peers { permanent_uuid: "97509c025dbe428fabba21071efa72ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37461 } } peers { permanent_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43817 } }
I20260925 16:33:00.411106 2648 leader_election.cc:288] T 47c334f57049409294e227c6e26311e8 P 97509c025dbe428fabba21071efa72ce [CANDIDATE]: Term 1 election: Requested vote from peers 435b9fb0fe93430face8f32e4de081b4 (127.0.98.1:43805), a3d3aa90f8e74bb2ae128513c9f0523a (127.0.98.3:43817)
I20260925 16:33:00.411432 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c334f57049409294e227c6e26311e8" candidate_uuid: "97509c025dbe428fabba21071efa72ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "435b9fb0fe93430face8f32e4de081b4"
I20260925 16:33:00.411438 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c334f57049409294e227c6e26311e8" candidate_uuid: "97509c025dbe428fabba21071efa72ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a"
I20260925 16:33:00.411567 2293 raft_consensus.cc:3037] T 47c334f57049409294e227c6e26311e8 P 435b9fb0fe93430face8f32e4de081b4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:33:00.411563 2572 raft_consensus.cc:3037] T 47c334f57049409294e227c6e26311e8 P a3d3aa90f8e74bb2ae128513c9f0523a [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:33:00.412731 2293 raft_consensus.cc:2454] T 47c334f57049409294e227c6e26311e8 P 435b9fb0fe93430face8f32e4de081b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97509c025dbe428fabba21071efa72ce in term 1.
I20260925 16:33:00.412731 2572 raft_consensus.cc:2454] T 47c334f57049409294e227c6e26311e8 P a3d3aa90f8e74bb2ae128513c9f0523a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97509c025dbe428fabba21071efa72ce in term 1.
I20260925 16:33:00.413033 2365 leader_election.cc:302] T 47c334f57049409294e227c6e26311e8 P 97509c025dbe428fabba21071efa72ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 435b9fb0fe93430face8f32e4de081b4, 97509c025dbe428fabba21071efa72ce; no voters:
I20260925 16:33:00.413211 2648 raft_consensus.cc:2781] T 47c334f57049409294e227c6e26311e8 P 97509c025dbe428fabba21071efa72ce [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:33:00.413436 2648 raft_consensus.cc:686] T 47c334f57049409294e227c6e26311e8 P 97509c025dbe428fabba21071efa72ce [term 1 LEADER]: Becoming Leader. State: Replica: 97509c025dbe428fabba21071efa72ce, State: Running, Role: LEADER
I20260925 16:33:00.413729 2648 consensus_queue.cc:237] T 47c334f57049409294e227c6e26311e8 P 97509c025dbe428fabba21071efa72ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435b9fb0fe93430face8f32e4de081b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43805 } } peers { permanent_uuid: "97509c025dbe428fabba21071efa72ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37461 } } peers { permanent_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43817 } }
I20260925 16:33:00.416249 2136 catalog_manager.cc:5495] T 47c334f57049409294e227c6e26311e8 P 97509c025dbe428fabba21071efa72ce reported cstate change: term changed from 0 to 1, leader changed from <none> to 97509c025dbe428fabba21071efa72ce (127.0.98.2). New cstate: current_term: 1 leader_uuid: "97509c025dbe428fabba21071efa72ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435b9fb0fe93430face8f32e4de081b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97509c025dbe428fabba21071efa72ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43817 } health_report { overall_health: UNKNOWN } } }
I20260925 16:33:00.448565 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:00.444720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37932 (local address 127.0.98.2:37461)
0925 16:33:00.444858 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:00.444865 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:00.444882 (+ 17us) server_negotiation.cc:407] Connection header received
0925 16:33:00.445111 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:00.445114 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:00.445181 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:00.445303 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:33:00.445560 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:00.446077 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:00.447000 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:00.447226 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:00.447345 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:00.447657 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:33:00.448129 (+ 472us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:33:00.448262 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:00.448341 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 16:33:00.448398 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 16:33:00.458953 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:00.455635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58377 (local address 127.0.98.1:43805)
0925 16:33:00.455772 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:00.455777 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:00.455919 (+ 142us) server_negotiation.cc:407] Connection header received
0925 16:33:00.456074 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:00.456077 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:00.456127 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:00.456211 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:33:00.456461 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:00.456950 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:00.457835 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:00.458066 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:00.458173 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:00.458473 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:33:00.458538 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:33:00.458662 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:00.458763 (+ 101us) server_negotiation.cc:299] Negotiation successful
0925 16:33:00.458812 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 16:33:00.458994 2703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:00.455488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43805 (local address 127.0.98.2:58377)
0925 16:33:00.455840 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:00.455852 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:00.455966 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:00.456222 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:00.456225 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:00.456233 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 16:33:00.456356 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:00.456362 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:00.457061 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:00.457066 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:00.457695 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:00.457703 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:00.458186 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:00.458190 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:00.458291 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:00.458331 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 16:33:00.458683 (+ 352us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 16:33:00.458690 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 16:33:00.458755 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 16:33:00.458840 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":124,"threads_started":1}
I20260925 16:33:00.464713 2572 raft_consensus.cc:1261] T 47c334f57049409294e227c6e26311e8 P a3d3aa90f8e74bb2ae128513c9f0523a [term 1 FOLLOWER]: Refusing update from remote peer 97509c025dbe428fabba21071efa72ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:33:00.464735 2293 raft_consensus.cc:1261] T 47c334f57049409294e227c6e26311e8 P 435b9fb0fe93430face8f32e4de081b4 [term 1 FOLLOWER]: Refusing update from remote peer 97509c025dbe428fabba21071efa72ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:33:00.465157 2648 consensus_queue.cc:1035] T 47c334f57049409294e227c6e26311e8 P 97509c025dbe428fabba21071efa72ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "435b9fb0fe93430face8f32e4de081b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:33:00.465376 2699 consensus_queue.cc:1035] T 47c334f57049409294e227c6e26311e8 P 97509c025dbe428fabba21071efa72ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:33:00.482326 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:00.479293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45181 (local address 127.0.98.2:37461)
0925 16:33:00.479413 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:00.479417 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:00.479465 (+ 48us) server_negotiation.cc:407] Connection header received
0925 16:33:00.479580 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:00.479582 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:00.479644 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:00.479723 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:33:00.479972 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:00.480574 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:00.481348 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:00.481544 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:00.481625 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:00.481903 (+ 278us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:33:00.481981 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:33:00.482060 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:00.482146 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 16:33:00.482191 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 16:33:00.482326 2688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:00.479237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37461 (local address 127.0.98.1:45181)
0925 16:33:00.479415 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:00.479427 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:00.479500 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:00.479746 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:00.479748 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:00.479755 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 16:33:00.479864 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:00.479870 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:00.480680 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:00.480683 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:00.481252 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:00.481260 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:00.481643 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:00.481647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:00.481724 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:00.481764 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 16:33:00.482065 (+ 301us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 16:33:00.482074 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 16:33:00.482137 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 16:33:00.482212 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94}
W20260925 16:33:00.483026 2671 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 16:33:02.040745 2135 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:33:02.040892 2135 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:33:02.043473 2135 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2325bb98933b48e880649d64cbdac636: Sending DeleteTablet for 3 replicas of tablet 47c334f57049409294e227c6e26311e8
I20260925 16:33:02.044056 2273 tablet_service.cc:1514] Processing DeleteTablet for tablet 47c334f57049409294e227c6e26311e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:33:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58388
I20260925 16:33:02.044073 2413 tablet_service.cc:1514] Processing DeleteTablet for tablet 47c334f57049409294e227c6e26311e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:33:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37916
I20260925 16:33:02.044232 2553 tablet_service.cc:1514] Processing DeleteTablet for tablet 47c334f57049409294e227c6e26311e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:33:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41490
I20260925 16:33:02.044407 2733 tablet_replica.cc:330] T 47c334f57049409294e227c6e26311e8 P 435b9fb0fe93430face8f32e4de081b4: stopping tablet replica
I20260925 16:33:02.044535 2733 raft_consensus.cc:2229] T 47c334f57049409294e227c6e26311e8 P 435b9fb0fe93430face8f32e4de081b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:33:02.044653 2733 raft_consensus.cc:2258] T 47c334f57049409294e227c6e26311e8 P 435b9fb0fe93430face8f32e4de081b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:33:02.044700 2735 tablet_replica.cc:330] T 47c334f57049409294e227c6e26311e8 P a3d3aa90f8e74bb2ae128513c9f0523a: stopping tablet replica
I20260925 16:33:02.045037 2735 raft_consensus.cc:2229] T 47c334f57049409294e227c6e26311e8 P a3d3aa90f8e74bb2ae128513c9f0523a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:33:02.045186 2735 raft_consensus.cc:2258] T 47c334f57049409294e227c6e26311e8 P a3d3aa90f8e74bb2ae128513c9f0523a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:33:02.045426 2734 tablet_replica.cc:330] T 47c334f57049409294e227c6e26311e8 P 97509c025dbe428fabba21071efa72ce: stopping tablet replica
I20260925 16:33:02.045549 2734 raft_consensus.cc:2229] T 47c334f57049409294e227c6e26311e8 P 97509c025dbe428fabba21071efa72ce [term 1 LEADER]: Raft consensus shutting down.
I20260925 16:33:02.045691 2733 ts_tablet_manager.cc:1905] T 47c334f57049409294e227c6e26311e8 P 435b9fb0fe93430face8f32e4de081b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:33:02.045886 2734 raft_consensus.cc:2258] T 47c334f57049409294e227c6e26311e8 P 97509c025dbe428fabba21071efa72ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:33:02.046443 2735 ts_tablet_manager.cc:1905] T 47c334f57049409294e227c6e26311e8 P a3d3aa90f8e74bb2ae128513c9f0523a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:33:02.048858 2735 ts_tablet_manager.cc:1918] T 47c334f57049409294e227c6e26311e8 P a3d3aa90f8e74bb2ae128513c9f0523a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 16:33:02.048964 2735 log.cc:1196] T 47c334f57049409294e227c6e26311e8 P a3d3aa90f8e74bb2ae128513c9f0523a: Deleting WAL directory at /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/ts-2/wal/wals/47c334f57049409294e227c6e26311e8
I20260925 16:33:02.049077 2733 ts_tablet_manager.cc:1918] T 47c334f57049409294e227c6e26311e8 P 435b9fb0fe93430face8f32e4de081b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 16:33:02.049259 2735 ts_tablet_manager.cc:1939] T 47c334f57049409294e227c6e26311e8 P a3d3aa90f8e74bb2ae128513c9f0523a: Deleting consensus metadata
I20260925 16:33:02.049250 2733 log.cc:1196] T 47c334f57049409294e227c6e26311e8 P 435b9fb0fe93430face8f32e4de081b4: Deleting WAL directory at /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/ts-0/wal/wals/47c334f57049409294e227c6e26311e8
I20260925 16:33:02.049559 2733 ts_tablet_manager.cc:1939] T 47c334f57049409294e227c6e26311e8 P 435b9fb0fe93430face8f32e4de081b4: Deleting consensus metadata
I20260925 16:33:02.049829 2734 ts_tablet_manager.cc:1905] T 47c334f57049409294e227c6e26311e8 P 97509c025dbe428fabba21071efa72ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:33:02.050374 2121 catalog_manager.cc:4841] TS 435b9fb0fe93430face8f32e4de081b4 (127.0.98.1:43805): tablet 47c334f57049409294e227c6e26311e8 (table test-table [id=39970adee75643dca9b2a4909d0a29cf]) successfully deleted
I20260925 16:33:02.051218 2122 catalog_manager.cc:4841] TS a3d3aa90f8e74bb2ae128513c9f0523a (127.0.98.3:43817): tablet 47c334f57049409294e227c6e26311e8 (table test-table [id=39970adee75643dca9b2a4909d0a29cf]) successfully deleted
I20260925 16:33:02.052187 2734 ts_tablet_manager.cc:1918] T 47c334f57049409294e227c6e26311e8 P 97509c025dbe428fabba21071efa72ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 16:33:02.052253 2734 log.cc:1196] T 47c334f57049409294e227c6e26311e8 P 97509c025dbe428fabba21071efa72ce: Deleting WAL directory at /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/ts-1/wal/wals/47c334f57049409294e227c6e26311e8
I20260925 16:33:02.052515 2734 ts_tablet_manager.cc:1939] T 47c334f57049409294e227c6e26311e8 P 97509c025dbe428fabba21071efa72ce: Deleting consensus metadata
I20260925 16:33:02.053100 2123 catalog_manager.cc:4841] TS 97509c025dbe428fabba21071efa72ce (127.0.98.2:37461): tablet 47c334f57049409294e227c6e26311e8 (table test-table [id=39970adee75643dca9b2a4909d0a29cf]) successfully deleted
Sep 25 16:33:02 dist-test-slave-65pf krb5kdc[2091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353982, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 16:33:02 dist-test-slave-65pf krb5kdc[2091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353982, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:33:02.076346 2743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:02.062211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55678 (local address 127.0.98.62:45895)
0925 16:33:02.062432 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:02.062439 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:02.062514 (+ 75us) server_negotiation.cc:407] Connection header received
0925 16:33:02.062686 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:02.062688 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:02.062730 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:02.062805 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:02.063426 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:02.063905 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:02.064671 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:02.064881 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:02.064961 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:02.073636 (+ 8675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:02.073639 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:02.073641 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:02.073668 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:02.075002 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:02.075369 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:02.075372 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:02.075373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:02.075455 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:02.075671 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:02.075673 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:02.075674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:02.075841 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:02.075936 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:02.076128 (+ 192us) server_negotiation.cc:299] Negotiation successful
0925 16:33:02.076222 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":94,"threads_started":1}
I20260925 16:33:02.078800 2135 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55678:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 16:33:02.079260 2135 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:33:02.083724 2273 tablet_service.cc:1467] Processing CreateTablet for tablet 66f80d8370d044608da5dc9812a7826e (DEFAULT_TABLE table=test-table [id=8cc2548fb9f24d0fa383296284618797]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:33:02.083868 2413 tablet_service.cc:1467] Processing CreateTablet for tablet 66f80d8370d044608da5dc9812a7826e (DEFAULT_TABLE table=test-table [id=8cc2548fb9f24d0fa383296284618797]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:33:02.083942 2553 tablet_service.cc:1467] Processing CreateTablet for tablet 66f80d8370d044608da5dc9812a7826e (DEFAULT_TABLE table=test-table [id=8cc2548fb9f24d0fa383296284618797]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:33:02.084065 2273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66f80d8370d044608da5dc9812a7826e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:33:02.084156 2413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66f80d8370d044608da5dc9812a7826e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:33:02.084178 2553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66f80d8370d044608da5dc9812a7826e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:33:02.086292 2744 tablet_bootstrap.cc:492] T 66f80d8370d044608da5dc9812a7826e P 435b9fb0fe93430face8f32e4de081b4: Bootstrap starting.
I20260925 16:33:02.086903 2745 tablet_bootstrap.cc:492] T 66f80d8370d044608da5dc9812a7826e P 97509c025dbe428fabba21071efa72ce: Bootstrap starting.
I20260925 16:33:02.087282 2746 tablet_bootstrap.cc:492] T 66f80d8370d044608da5dc9812a7826e P a3d3aa90f8e74bb2ae128513c9f0523a: Bootstrap starting.
I20260925 16:33:02.087303 2744 tablet_bootstrap.cc:654] T 66f80d8370d044608da5dc9812a7826e P 435b9fb0fe93430face8f32e4de081b4: Neither blocks nor log segments found. Creating new log.
I20260925 16:33:02.087792 2745 tablet_bootstrap.cc:654] T 66f80d8370d044608da5dc9812a7826e P 97509c025dbe428fabba21071efa72ce: Neither blocks nor log segments found. Creating new log.
I20260925 16:33:02.088145 2746 tablet_bootstrap.cc:654] T 66f80d8370d044608da5dc9812a7826e P a3d3aa90f8e74bb2ae128513c9f0523a: Neither blocks nor log segments found. Creating new log.
I20260925 16:33:02.088523 2744 tablet_bootstrap.cc:492] T 66f80d8370d044608da5dc9812a7826e P 435b9fb0fe93430face8f32e4de081b4: No bootstrap required, opened a new log
I20260925 16:33:02.088625 2744 ts_tablet_manager.cc:1397] T 66f80d8370d044608da5dc9812a7826e P 435b9fb0fe93430face8f32e4de081b4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 16:33:02.088932 2745 tablet_bootstrap.cc:492] T 66f80d8370d044608da5dc9812a7826e P 97509c025dbe428fabba21071efa72ce: No bootstrap required, opened a new log
I20260925 16:33:02.089018 2745 ts_tablet_manager.cc:1397] T 66f80d8370d044608da5dc9812a7826e P 97509c025dbe428fabba21071efa72ce: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 16:33:02.089134 2744 raft_consensus.cc:348] T 66f80d8370d044608da5dc9812a7826e P 435b9fb0fe93430face8f32e4de081b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435b9fb0fe93430face8f32e4de081b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43805 } } peers { permanent_uuid: "97509c025dbe428fabba21071efa72ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37461 } } peers { permanent_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43817 } }
I20260925 16:33:02.089242 2744 raft_consensus.cc:374] T 66f80d8370d044608da5dc9812a7826e P 435b9fb0fe93430face8f32e4de081b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:33:02.089262 2746 tablet_bootstrap.cc:492] T 66f80d8370d044608da5dc9812a7826e P a3d3aa90f8e74bb2ae128513c9f0523a: No bootstrap required, opened a new log
I20260925 16:33:02.089278 2744 raft_consensus.cc:729] T 66f80d8370d044608da5dc9812a7826e P 435b9fb0fe93430face8f32e4de081b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 435b9fb0fe93430face8f32e4de081b4, State: Initialized, Role: FOLLOWER
I20260925 16:33:02.089336 2746 ts_tablet_manager.cc:1397] T 66f80d8370d044608da5dc9812a7826e P a3d3aa90f8e74bb2ae128513c9f0523a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 16:33:02.089428 2744 consensus_queue.cc:260] T 66f80d8370d044608da5dc9812a7826e P 435b9fb0fe93430face8f32e4de081b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435b9fb0fe93430face8f32e4de081b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43805 } } peers { permanent_uuid: "97509c025dbe428fabba21071efa72ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37461 } } peers { permanent_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43817 } }
I20260925 16:33:02.089450 2745 raft_consensus.cc:348] T 66f80d8370d044608da5dc9812a7826e P 97509c025dbe428fabba21071efa72ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435b9fb0fe93430face8f32e4de081b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43805 } } peers { permanent_uuid: "97509c025dbe428fabba21071efa72ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37461 } } peers { permanent_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43817 } }
I20260925 16:33:02.089556 2745 raft_consensus.cc:374] T 66f80d8370d044608da5dc9812a7826e P 97509c025dbe428fabba21071efa72ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:33:02.089596 2745 raft_consensus.cc:729] T 66f80d8370d044608da5dc9812a7826e P 97509c025dbe428fabba21071efa72ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97509c025dbe428fabba21071efa72ce, State: Initialized, Role: FOLLOWER
I20260925 16:33:02.089711 2745 consensus_queue.cc:260] T 66f80d8370d044608da5dc9812a7826e P 97509c025dbe428fabba21071efa72ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435b9fb0fe93430face8f32e4de081b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43805 } } peers { permanent_uuid: "97509c025dbe428fabba21071efa72ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37461 } } peers { permanent_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43817 } }
I20260925 16:33:02.089722 2746 raft_consensus.cc:348] T 66f80d8370d044608da5dc9812a7826e P a3d3aa90f8e74bb2ae128513c9f0523a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435b9fb0fe93430face8f32e4de081b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43805 } } peers { permanent_uuid: "97509c025dbe428fabba21071efa72ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37461 } } peers { permanent_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43817 } }
I20260925 16:33:02.089819 2746 raft_consensus.cc:374] T 66f80d8370d044608da5dc9812a7826e P a3d3aa90f8e74bb2ae128513c9f0523a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:33:02.089860 2746 raft_consensus.cc:729] T 66f80d8370d044608da5dc9812a7826e P a3d3aa90f8e74bb2ae128513c9f0523a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3d3aa90f8e74bb2ae128513c9f0523a, State: Initialized, Role: FOLLOWER
I20260925 16:33:02.089993 2745 ts_tablet_manager.cc:1428] T 66f80d8370d044608da5dc9812a7826e P 97509c025dbe428fabba21071efa72ce: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 16:33:02.089982 2746 consensus_queue.cc:260] T 66f80d8370d044608da5dc9812a7826e P a3d3aa90f8e74bb2ae128513c9f0523a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435b9fb0fe93430face8f32e4de081b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43805 } } peers { permanent_uuid: "97509c025dbe428fabba21071efa72ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37461 } } peers { permanent_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43817 } }
I20260925 16:33:02.090334 2746 ts_tablet_manager.cc:1428] T 66f80d8370d044608da5dc9812a7826e P a3d3aa90f8e74bb2ae128513c9f0523a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 16:33:02.090670 2744 ts_tablet_manager.cc:1428] T 66f80d8370d044608da5dc9812a7826e P 435b9fb0fe93430face8f32e4de081b4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 16:33:02.105574 2710 raft_consensus.cc:482] T 66f80d8370d044608da5dc9812a7826e P 97509c025dbe428fabba21071efa72ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:33:02.105698 2710 raft_consensus.cc:504] T 66f80d8370d044608da5dc9812a7826e P 97509c025dbe428fabba21071efa72ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435b9fb0fe93430face8f32e4de081b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43805 } } peers { permanent_uuid: "97509c025dbe428fabba21071efa72ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37461 } } peers { permanent_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43817 } }
I20260925 16:33:02.106325 2710 leader_election.cc:288] T 66f80d8370d044608da5dc9812a7826e P 97509c025dbe428fabba21071efa72ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 435b9fb0fe93430face8f32e4de081b4 (127.0.98.1:43805), a3d3aa90f8e74bb2ae128513c9f0523a (127.0.98.3:43817)
I20260925 16:33:02.106776 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66f80d8370d044608da5dc9812a7826e" candidate_uuid: "97509c025dbe428fabba21071efa72ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "435b9fb0fe93430face8f32e4de081b4" is_pre_election: true
I20260925 16:33:02.106863 2573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66f80d8370d044608da5dc9812a7826e" candidate_uuid: "97509c025dbe428fabba21071efa72ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" is_pre_election: true
I20260925 16:33:02.106951 2293 raft_consensus.cc:2454] T 66f80d8370d044608da5dc9812a7826e P 435b9fb0fe93430face8f32e4de081b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97509c025dbe428fabba21071efa72ce in term 0.
I20260925 16:33:02.107045 2573 raft_consensus.cc:2454] T 66f80d8370d044608da5dc9812a7826e P a3d3aa90f8e74bb2ae128513c9f0523a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97509c025dbe428fabba21071efa72ce in term 0.
I20260925 16:33:02.107302 2365 leader_election.cc:302] T 66f80d8370d044608da5dc9812a7826e P 97509c025dbe428fabba21071efa72ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 435b9fb0fe93430face8f32e4de081b4, 97509c025dbe428fabba21071efa72ce; no voters:
I20260925 16:33:02.107554 2710 raft_consensus.cc:2781] T 66f80d8370d044608da5dc9812a7826e P 97509c025dbe428fabba21071efa72ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:33:02.107656 2710 raft_consensus.cc:482] T 66f80d8370d044608da5dc9812a7826e P 97509c025dbe428fabba21071efa72ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:33:02.107726 2710 raft_consensus.cc:3037] T 66f80d8370d044608da5dc9812a7826e P 97509c025dbe428fabba21071efa72ce [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:33:02.108875 2710 raft_consensus.cc:504] T 66f80d8370d044608da5dc9812a7826e P 97509c025dbe428fabba21071efa72ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435b9fb0fe93430face8f32e4de081b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43805 } } peers { permanent_uuid: "97509c025dbe428fabba21071efa72ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37461 } } peers { permanent_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43817 } }
I20260925 16:33:02.109335 2710 leader_election.cc:288] T 66f80d8370d044608da5dc9812a7826e P 97509c025dbe428fabba21071efa72ce [CANDIDATE]: Term 1 election: Requested vote from peers 435b9fb0fe93430face8f32e4de081b4 (127.0.98.1:43805), a3d3aa90f8e74bb2ae128513c9f0523a (127.0.98.3:43817)
I20260925 16:33:02.109678 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66f80d8370d044608da5dc9812a7826e" candidate_uuid: "97509c025dbe428fabba21071efa72ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "435b9fb0fe93430face8f32e4de081b4"
I20260925 16:33:02.109733 2573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66f80d8370d044608da5dc9812a7826e" candidate_uuid: "97509c025dbe428fabba21071efa72ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a"
I20260925 16:33:02.109818 2293 raft_consensus.cc:3037] T 66f80d8370d044608da5dc9812a7826e P 435b9fb0fe93430face8f32e4de081b4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:33:02.109840 2573 raft_consensus.cc:3037] T 66f80d8370d044608da5dc9812a7826e P a3d3aa90f8e74bb2ae128513c9f0523a [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:33:02.111002 2293 raft_consensus.cc:2454] T 66f80d8370d044608da5dc9812a7826e P 435b9fb0fe93430face8f32e4de081b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97509c025dbe428fabba21071efa72ce in term 1.
I20260925 16:33:02.111002 2573 raft_consensus.cc:2454] T 66f80d8370d044608da5dc9812a7826e P a3d3aa90f8e74bb2ae128513c9f0523a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97509c025dbe428fabba21071efa72ce in term 1.
I20260925 16:33:02.111289 2366 leader_election.cc:302] T 66f80d8370d044608da5dc9812a7826e P 97509c025dbe428fabba21071efa72ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97509c025dbe428fabba21071efa72ce, a3d3aa90f8e74bb2ae128513c9f0523a; no voters:
I20260925 16:33:02.111485 2710 raft_consensus.cc:2781] T 66f80d8370d044608da5dc9812a7826e P 97509c025dbe428fabba21071efa72ce [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:33:02.111614 2710 raft_consensus.cc:686] T 66f80d8370d044608da5dc9812a7826e P 97509c025dbe428fabba21071efa72ce [term 1 LEADER]: Becoming Leader. State: Replica: 97509c025dbe428fabba21071efa72ce, State: Running, Role: LEADER
I20260925 16:33:02.111795 2710 consensus_queue.cc:237] T 66f80d8370d044608da5dc9812a7826e P 97509c025dbe428fabba21071efa72ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435b9fb0fe93430face8f32e4de081b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43805 } } peers { permanent_uuid: "97509c025dbe428fabba21071efa72ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37461 } } peers { permanent_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43817 } }
I20260925 16:33:02.113447 2134 catalog_manager.cc:5495] T 66f80d8370d044608da5dc9812a7826e P 97509c025dbe428fabba21071efa72ce reported cstate change: term changed from 0 to 1, leader changed from <none> to 97509c025dbe428fabba21071efa72ce (127.0.98.2). New cstate: current_term: 1 leader_uuid: "97509c025dbe428fabba21071efa72ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435b9fb0fe93430face8f32e4de081b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97509c025dbe428fabba21071efa72ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43817 } health_report { overall_health: UNKNOWN } } }
I20260925 16:33:02.127928 2753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:02.124938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37940 (local address 127.0.98.2:37461)
0925 16:33:02.125182 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:02.125189 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:02.125200 (+ 11us) server_negotiation.cc:407] Connection header received
0925 16:33:02.125238 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:02.125240 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:02.125277 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:02.125346 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:33:02.125548 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:02.126033 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:02.126908 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:02.127133 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:02.127206 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:02.127443 (+ 237us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:33:02.127506 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:33:02.127558 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:02.127633 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 16:33:02.127672 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":112,"threads_started":1}
I20260925 16:33:02.133455 2293 raft_consensus.cc:1261] T 66f80d8370d044608da5dc9812a7826e P 435b9fb0fe93430face8f32e4de081b4 [term 1 FOLLOWER]: Refusing update from remote peer 97509c025dbe428fabba21071efa72ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:33:02.133469 2573 raft_consensus.cc:1261] T 66f80d8370d044608da5dc9812a7826e P a3d3aa90f8e74bb2ae128513c9f0523a [term 1 FOLLOWER]: Refusing update from remote peer 97509c025dbe428fabba21071efa72ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:33:02.133805 2723 consensus_queue.cc:1035] T 66f80d8370d044608da5dc9812a7826e P 97509c025dbe428fabba21071efa72ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "435b9fb0fe93430face8f32e4de081b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:33:02.133948 2710 consensus_queue.cc:1035] T 66f80d8370d044608da5dc9812a7826e P 97509c025dbe428fabba21071efa72ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:33:02.162847 2136 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:33:02.162984 2136 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:33:02.164695 2136 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2325bb98933b48e880649d64cbdac636: Sending DeleteTablet for 3 replicas of tablet 66f80d8370d044608da5dc9812a7826e
I20260925 16:33:02.165226 2273 tablet_service.cc:1514] Processing DeleteTablet for tablet 66f80d8370d044608da5dc9812a7826e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:33:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58388
I20260925 16:33:02.165203 2413 tablet_service.cc:1514] Processing DeleteTablet for tablet 66f80d8370d044608da5dc9812a7826e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:33:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37916
I20260925 16:33:02.165410 2734 tablet_replica.cc:330] T 66f80d8370d044608da5dc9812a7826e P 97509c025dbe428fabba21071efa72ce: stopping tablet replica
I20260925 16:33:02.165530 2734 raft_consensus.cc:2229] T 66f80d8370d044608da5dc9812a7826e P 97509c025dbe428fabba21071efa72ce [term 1 LEADER]: Raft consensus shutting down.
I20260925 16:33:02.165704 2733 tablet_replica.cc:330] T 66f80d8370d044608da5dc9812a7826e P 435b9fb0fe93430face8f32e4de081b4: stopping tablet replica
I20260925 16:33:02.165700 2734 raft_consensus.cc:2258] T 66f80d8370d044608da5dc9812a7826e P 97509c025dbe428fabba21071efa72ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:33:02.165832 2733 raft_consensus.cc:2229] T 66f80d8370d044608da5dc9812a7826e P 435b9fb0fe93430face8f32e4de081b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:33:02.165936 2733 raft_consensus.cc:2258] T 66f80d8370d044608da5dc9812a7826e P 435b9fb0fe93430face8f32e4de081b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:33:02.166332 2734 ts_tablet_manager.cc:1905] T 66f80d8370d044608da5dc9812a7826e P 97509c025dbe428fabba21071efa72ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:33:02.166654 2733 ts_tablet_manager.cc:1905] T 66f80d8370d044608da5dc9812a7826e P 435b9fb0fe93430face8f32e4de081b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:33:02.168555 2734 ts_tablet_manager.cc:1918] T 66f80d8370d044608da5dc9812a7826e P 97509c025dbe428fabba21071efa72ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 16:33:02.168606 2734 log.cc:1196] T 66f80d8370d044608da5dc9812a7826e P 97509c025dbe428fabba21071efa72ce: Deleting WAL directory at /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/ts-1/wal/wals/66f80d8370d044608da5dc9812a7826e
I20260925 16:33:02.168731 2733 ts_tablet_manager.cc:1918] T 66f80d8370d044608da5dc9812a7826e P 435b9fb0fe93430face8f32e4de081b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 16:33:02.168788 2733 log.cc:1196] T 66f80d8370d044608da5dc9812a7826e P 435b9fb0fe93430face8f32e4de081b4: Deleting WAL directory at /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/ts-0/wal/wals/66f80d8370d044608da5dc9812a7826e
I20260925 16:33:02.168803 2734 ts_tablet_manager.cc:1939] T 66f80d8370d044608da5dc9812a7826e P 97509c025dbe428fabba21071efa72ce: Deleting consensus metadata
I20260925 16:33:02.168968 2733 ts_tablet_manager.cc:1939] T 66f80d8370d044608da5dc9812a7826e P 435b9fb0fe93430face8f32e4de081b4: Deleting consensus metadata
I20260925 16:33:02.168951 2553 tablet_service.cc:1514] Processing DeleteTablet for tablet 66f80d8370d044608da5dc9812a7826e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:33:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41490
I20260925 16:33:02.169165 2735 tablet_replica.cc:330] T 66f80d8370d044608da5dc9812a7826e P a3d3aa90f8e74bb2ae128513c9f0523a: stopping tablet replica
I20260925 16:33:02.169278 2735 raft_consensus.cc:2229] T 66f80d8370d044608da5dc9812a7826e P a3d3aa90f8e74bb2ae128513c9f0523a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:33:02.169379 2735 raft_consensus.cc:2258] T 66f80d8370d044608da5dc9812a7826e P a3d3aa90f8e74bb2ae128513c9f0523a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:33:02.169404 2136 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55678:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 16:33:02.169659 2123 catalog_manager.cc:4841] TS 97509c025dbe428fabba21071efa72ce (127.0.98.2:37461): tablet 66f80d8370d044608da5dc9812a7826e (table test-table [id=8cc2548fb9f24d0fa383296284618797]) successfully deleted
I20260925 16:33:02.169792 2121 catalog_manager.cc:4841] TS 435b9fb0fe93430face8f32e4de081b4 (127.0.98.1:43805): tablet 66f80d8370d044608da5dc9812a7826e (table test-table [id=8cc2548fb9f24d0fa383296284618797]) successfully deleted
W20260925 16:33:02.169835 2136 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:33:02.170047 2735 ts_tablet_manager.cc:1905] T 66f80d8370d044608da5dc9812a7826e P a3d3aa90f8e74bb2ae128513c9f0523a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:33:02.172561 2735 ts_tablet_manager.cc:1918] T 66f80d8370d044608da5dc9812a7826e P a3d3aa90f8e74bb2ae128513c9f0523a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 16:33:02.172642 2735 log.cc:1196] T 66f80d8370d044608da5dc9812a7826e P a3d3aa90f8e74bb2ae128513c9f0523a: Deleting WAL directory at /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/ts-2/wal/wals/66f80d8370d044608da5dc9812a7826e
I20260925 16:33:02.172921 2735 ts_tablet_manager.cc:1939] T 66f80d8370d044608da5dc9812a7826e P a3d3aa90f8e74bb2ae128513c9f0523a: Deleting consensus metadata
I20260925 16:33:02.173596 2122 catalog_manager.cc:4841] TS a3d3aa90f8e74bb2ae128513c9f0523a (127.0.98.3:43817): tablet 66f80d8370d044608da5dc9812a7826e (table test-table [id=8cc2548fb9f24d0fa383296284618797]) successfully deleted
I20260925 16:33:02.174392 2273 tablet_service.cc:1467] Processing CreateTablet for tablet 3d764d0b4ace4b6f90b529acb9ef8d72 (DEFAULT_TABLE table=test-table [id=44c2fe7773d146839f003f618fad18a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:33:02.174405 2413 tablet_service.cc:1467] Processing CreateTablet for tablet 3d764d0b4ace4b6f90b529acb9ef8d72 (DEFAULT_TABLE table=test-table [id=44c2fe7773d146839f003f618fad18a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:33:02.174543 2553 tablet_service.cc:1467] Processing CreateTablet for tablet 3d764d0b4ace4b6f90b529acb9ef8d72 (DEFAULT_TABLE table=test-table [id=44c2fe7773d146839f003f618fad18a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:33:02.174670 2413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d764d0b4ace4b6f90b529acb9ef8d72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:33:02.174669 2273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d764d0b4ace4b6f90b529acb9ef8d72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:33:02.174798 2553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d764d0b4ace4b6f90b529acb9ef8d72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:33:02.176340 2744 tablet_bootstrap.cc:492] T 3d764d0b4ace4b6f90b529acb9ef8d72 P 435b9fb0fe93430face8f32e4de081b4: Bootstrap starting.
I20260925 16:33:02.176779 2746 tablet_bootstrap.cc:492] T 3d764d0b4ace4b6f90b529acb9ef8d72 P a3d3aa90f8e74bb2ae128513c9f0523a: Bootstrap starting.
I20260925 16:33:02.176817 2745 tablet_bootstrap.cc:492] T 3d764d0b4ace4b6f90b529acb9ef8d72 P 97509c025dbe428fabba21071efa72ce: Bootstrap starting.
I20260925 16:33:02.177177 2744 tablet_bootstrap.cc:654] T 3d764d0b4ace4b6f90b529acb9ef8d72 P 435b9fb0fe93430face8f32e4de081b4: Neither blocks nor log segments found. Creating new log.
I20260925 16:33:02.177536 2745 tablet_bootstrap.cc:654] T 3d764d0b4ace4b6f90b529acb9ef8d72 P 97509c025dbe428fabba21071efa72ce: Neither blocks nor log segments found. Creating new log.
I20260925 16:33:02.177660 2746 tablet_bootstrap.cc:654] T 3d764d0b4ace4b6f90b529acb9ef8d72 P a3d3aa90f8e74bb2ae128513c9f0523a: Neither blocks nor log segments found. Creating new log.
I20260925 16:33:02.178393 2744 tablet_bootstrap.cc:492] T 3d764d0b4ace4b6f90b529acb9ef8d72 P 435b9fb0fe93430face8f32e4de081b4: No bootstrap required, opened a new log
I20260925 16:33:02.178503 2744 ts_tablet_manager.cc:1397] T 3d764d0b4ace4b6f90b529acb9ef8d72 P 435b9fb0fe93430face8f32e4de081b4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 16:33:02.178717 2745 tablet_bootstrap.cc:492] T 3d764d0b4ace4b6f90b529acb9ef8d72 P 97509c025dbe428fabba21071efa72ce: No bootstrap required, opened a new log
I20260925 16:33:02.178786 2745 ts_tablet_manager.cc:1397] T 3d764d0b4ace4b6f90b529acb9ef8d72 P 97509c025dbe428fabba21071efa72ce: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 16:33:02.178949 2744 raft_consensus.cc:348] T 3d764d0b4ace4b6f90b529acb9ef8d72 P 435b9fb0fe93430face8f32e4de081b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435b9fb0fe93430face8f32e4de081b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43805 } } peers { permanent_uuid: "97509c025dbe428fabba21071efa72ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37461 } } peers { permanent_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43817 } }
I20260925 16:33:02.179064 2744 raft_consensus.cc:374] T 3d764d0b4ace4b6f90b529acb9ef8d72 P 435b9fb0fe93430face8f32e4de081b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:33:02.179108 2744 raft_consensus.cc:729] T 3d764d0b4ace4b6f90b529acb9ef8d72 P 435b9fb0fe93430face8f32e4de081b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 435b9fb0fe93430face8f32e4de081b4, State: Initialized, Role: FOLLOWER
I20260925 16:33:02.179072 2745 raft_consensus.cc:348] T 3d764d0b4ace4b6f90b529acb9ef8d72 P 97509c025dbe428fabba21071efa72ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435b9fb0fe93430face8f32e4de081b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43805 } } peers { permanent_uuid: "97509c025dbe428fabba21071efa72ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37461 } } peers { permanent_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43817 } }
I20260925 16:33:02.179209 2745 raft_consensus.cc:374] T 3d764d0b4ace4b6f90b529acb9ef8d72 P 97509c025dbe428fabba21071efa72ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:33:02.179256 2745 raft_consensus.cc:729] T 3d764d0b4ace4b6f90b529acb9ef8d72 P 97509c025dbe428fabba21071efa72ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97509c025dbe428fabba21071efa72ce, State: Initialized, Role: FOLLOWER
I20260925 16:33:02.179229 2744 consensus_queue.cc:260] T 3d764d0b4ace4b6f90b529acb9ef8d72 P 435b9fb0fe93430face8f32e4de081b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435b9fb0fe93430face8f32e4de081b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43805 } } peers { permanent_uuid: "97509c025dbe428fabba21071efa72ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37461 } } peers { permanent_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43817 } }
I20260925 16:33:02.179399 2745 consensus_queue.cc:260] T 3d764d0b4ace4b6f90b529acb9ef8d72 P 97509c025dbe428fabba21071efa72ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435b9fb0fe93430face8f32e4de081b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43805 } } peers { permanent_uuid: "97509c025dbe428fabba21071efa72ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37461 } } peers { permanent_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43817 } }
I20260925 16:33:02.179477 2744 ts_tablet_manager.cc:1428] T 3d764d0b4ace4b6f90b529acb9ef8d72 P 435b9fb0fe93430face8f32e4de081b4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 16:33:02.179673 2745 ts_tablet_manager.cc:1428] T 3d764d0b4ace4b6f90b529acb9ef8d72 P 97509c025dbe428fabba21071efa72ce: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260925 16:33:02.180369 2746 tablet_bootstrap.cc:492] T 3d764d0b4ace4b6f90b529acb9ef8d72 P a3d3aa90f8e74bb2ae128513c9f0523a: No bootstrap required, opened a new log
I20260925 16:33:02.180471 2746 ts_tablet_manager.cc:1397] T 3d764d0b4ace4b6f90b529acb9ef8d72 P a3d3aa90f8e74bb2ae128513c9f0523a: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260925 16:33:02.180907 2746 raft_consensus.cc:348] T 3d764d0b4ace4b6f90b529acb9ef8d72 P a3d3aa90f8e74bb2ae128513c9f0523a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435b9fb0fe93430face8f32e4de081b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43805 } } peers { permanent_uuid: "97509c025dbe428fabba21071efa72ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37461 } } peers { permanent_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43817 } }
I20260925 16:33:02.181030 2746 raft_consensus.cc:374] T 3d764d0b4ace4b6f90b529acb9ef8d72 P a3d3aa90f8e74bb2ae128513c9f0523a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:33:02.181064 2746 raft_consensus.cc:729] T 3d764d0b4ace4b6f90b529acb9ef8d72 P a3d3aa90f8e74bb2ae128513c9f0523a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3d3aa90f8e74bb2ae128513c9f0523a, State: Initialized, Role: FOLLOWER
I20260925 16:33:02.181183 2746 consensus_queue.cc:260] T 3d764d0b4ace4b6f90b529acb9ef8d72 P a3d3aa90f8e74bb2ae128513c9f0523a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435b9fb0fe93430face8f32e4de081b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43805 } } peers { permanent_uuid: "97509c025dbe428fabba21071efa72ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37461 } } peers { permanent_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43817 } }
I20260925 16:33:02.181468 2746 ts_tablet_manager.cc:1428] T 3d764d0b4ace4b6f90b529acb9ef8d72 P a3d3aa90f8e74bb2ae128513c9f0523a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 16:33:02.329574 2750 raft_consensus.cc:482] T 3d764d0b4ace4b6f90b529acb9ef8d72 P a3d3aa90f8e74bb2ae128513c9f0523a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:33:02.329705 2750 raft_consensus.cc:504] T 3d764d0b4ace4b6f90b529acb9ef8d72 P a3d3aa90f8e74bb2ae128513c9f0523a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435b9fb0fe93430face8f32e4de081b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43805 } } peers { permanent_uuid: "97509c025dbe428fabba21071efa72ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37461 } } peers { permanent_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43817 } }
I20260925 16:33:02.330183 2750 leader_election.cc:288] T 3d764d0b4ace4b6f90b529acb9ef8d72 P a3d3aa90f8e74bb2ae128513c9f0523a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 435b9fb0fe93430face8f32e4de081b4 (127.0.98.1:43805), 97509c025dbe428fabba21071efa72ce (127.0.98.2:37461)
I20260925 16:33:02.330565 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d764d0b4ace4b6f90b529acb9ef8d72" candidate_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "435b9fb0fe93430face8f32e4de081b4" is_pre_election: true
I20260925 16:33:02.330736 2293 raft_consensus.cc:2454] T 3d764d0b4ace4b6f90b529acb9ef8d72 P 435b9fb0fe93430face8f32e4de081b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3d3aa90f8e74bb2ae128513c9f0523a in term 0.
I20260925 16:33:02.330729 2432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d764d0b4ace4b6f90b529acb9ef8d72" candidate_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97509c025dbe428fabba21071efa72ce" is_pre_election: true
I20260925 16:33:02.330914 2432 raft_consensus.cc:2454] T 3d764d0b4ace4b6f90b529acb9ef8d72 P 97509c025dbe428fabba21071efa72ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3d3aa90f8e74bb2ae128513c9f0523a in term 0.
I20260925 16:33:02.331107 2505 leader_election.cc:302] T 3d764d0b4ace4b6f90b529acb9ef8d72 P a3d3aa90f8e74bb2ae128513c9f0523a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 435b9fb0fe93430face8f32e4de081b4, a3d3aa90f8e74bb2ae128513c9f0523a; no voters:
I20260925 16:33:02.331300 2750 raft_consensus.cc:2781] T 3d764d0b4ace4b6f90b529acb9ef8d72 P a3d3aa90f8e74bb2ae128513c9f0523a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:33:02.331367 2750 raft_consensus.cc:482] T 3d764d0b4ace4b6f90b529acb9ef8d72 P a3d3aa90f8e74bb2ae128513c9f0523a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:33:02.331414 2750 raft_consensus.cc:3037] T 3d764d0b4ace4b6f90b529acb9ef8d72 P a3d3aa90f8e74bb2ae128513c9f0523a [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:33:02.332453 2750 raft_consensus.cc:504] T 3d764d0b4ace4b6f90b529acb9ef8d72 P a3d3aa90f8e74bb2ae128513c9f0523a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435b9fb0fe93430face8f32e4de081b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43805 } } peers { permanent_uuid: "97509c025dbe428fabba21071efa72ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37461 } } peers { permanent_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43817 } }
I20260925 16:33:02.332828 2750 leader_election.cc:288] T 3d764d0b4ace4b6f90b529acb9ef8d72 P a3d3aa90f8e74bb2ae128513c9f0523a [CANDIDATE]: Term 1 election: Requested vote from peers 435b9fb0fe93430face8f32e4de081b4 (127.0.98.1:43805), 97509c025dbe428fabba21071efa72ce (127.0.98.2:37461)
I20260925 16:33:02.333142 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d764d0b4ace4b6f90b529acb9ef8d72" candidate_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "435b9fb0fe93430face8f32e4de081b4"
I20260925 16:33:02.333256 2293 raft_consensus.cc:3037] T 3d764d0b4ace4b6f90b529acb9ef8d72 P 435b9fb0fe93430face8f32e4de081b4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:33:02.333212 2432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d764d0b4ace4b6f90b529acb9ef8d72" candidate_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97509c025dbe428fabba21071efa72ce"
I20260925 16:33:02.333371 2432 raft_consensus.cc:3037] T 3d764d0b4ace4b6f90b529acb9ef8d72 P 97509c025dbe428fabba21071efa72ce [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:33:02.334210 2293 raft_consensus.cc:2454] T 3d764d0b4ace4b6f90b529acb9ef8d72 P 435b9fb0fe93430face8f32e4de081b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3d3aa90f8e74bb2ae128513c9f0523a in term 1.
I20260925 16:33:02.334412 2432 raft_consensus.cc:2454] T 3d764d0b4ace4b6f90b529acb9ef8d72 P 97509c025dbe428fabba21071efa72ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3d3aa90f8e74bb2ae128513c9f0523a in term 1.
I20260925 16:33:02.334498 2505 leader_election.cc:302] T 3d764d0b4ace4b6f90b529acb9ef8d72 P a3d3aa90f8e74bb2ae128513c9f0523a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 435b9fb0fe93430face8f32e4de081b4, a3d3aa90f8e74bb2ae128513c9f0523a; no voters:
I20260925 16:33:02.334664 2750 raft_consensus.cc:2781] T 3d764d0b4ace4b6f90b529acb9ef8d72 P a3d3aa90f8e74bb2ae128513c9f0523a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:33:02.334872 2750 raft_consensus.cc:686] T 3d764d0b4ace4b6f90b529acb9ef8d72 P a3d3aa90f8e74bb2ae128513c9f0523a [term 1 LEADER]: Becoming Leader. State: Replica: a3d3aa90f8e74bb2ae128513c9f0523a, State: Running, Role: LEADER
I20260925 16:33:02.335029 2750 consensus_queue.cc:237] T 3d764d0b4ace4b6f90b529acb9ef8d72 P a3d3aa90f8e74bb2ae128513c9f0523a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435b9fb0fe93430face8f32e4de081b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43805 } } peers { permanent_uuid: "97509c025dbe428fabba21071efa72ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37461 } } peers { permanent_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43817 } }
I20260925 16:33:02.336550 2135 catalog_manager.cc:5495] T 3d764d0b4ace4b6f90b529acb9ef8d72 P a3d3aa90f8e74bb2ae128513c9f0523a reported cstate change: term changed from 0 to 1, leader changed from <none> to a3d3aa90f8e74bb2ae128513c9f0523a (127.0.98.3). New cstate: current_term: 1 leader_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435b9fb0fe93430face8f32e4de081b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97509c025dbe428fabba21071efa72ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3d3aa90f8e74bb2ae128513c9f0523a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43817 } health_report { overall_health: HEALTHY } } }
I20260925 16:33:02.376848 2762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:02.374100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41508 (local address 127.0.98.3:43817)
0925 16:33:02.374374 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:02.374380 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:02.374392 (+ 12us) server_negotiation.cc:407] Connection header received
0925 16:33:02.374426 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:02.374428 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:02.374463 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:02.374538 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:33:02.374771 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:02.375280 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:02.376007 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:02.376216 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:02.376304 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:02.376501 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:33:02.376569 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:33:02.376620 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:02.376698 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 16:33:02.376736 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":143,"threads_started":1}
I20260925 16:33:02.382949 2293 raft_consensus.cc:1261] T 3d764d0b4ace4b6f90b529acb9ef8d72 P 435b9fb0fe93430face8f32e4de081b4 [term 1 FOLLOWER]: Refusing update from remote peer a3d3aa90f8e74bb2ae128513c9f0523a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:33:02.383317 2432 raft_consensus.cc:1261] T 3d764d0b4ace4b6f90b529acb9ef8d72 P 97509c025dbe428fabba21071efa72ce [term 1 FOLLOWER]: Refusing update from remote peer a3d3aa90f8e74bb2ae128513c9f0523a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:33:02.383373 2750 consensus_queue.cc:1035] T 3d764d0b4ace4b6f90b529acb9ef8d72 P a3d3aa90f8e74bb2ae128513c9f0523a [LEADER]: Connected to new peer: Peer: permanent_uuid: "435b9fb0fe93430face8f32e4de081b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:33:02.383711 2750 consensus_queue.cc:1035] T 3d764d0b4ace4b6f90b529acb9ef8d72 P a3d3aa90f8e74bb2ae128513c9f0523a [LEADER]: Connected to new peer: Peer: permanent_uuid: "97509c025dbe428fabba21071efa72ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 16:33:02.395334 2714 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 16:33:03.953785 2135 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:33:03.953938 2135 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:33:03.956542 2135 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2325bb98933b48e880649d64cbdac636: Sending DeleteTablet for 3 replicas of tablet 3d764d0b4ace4b6f90b529acb9ef8d72
I20260925 16:33:03.957235 2273 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d764d0b4ace4b6f90b529acb9ef8d72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:33:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58388
I20260925 16:33:03.957264 2413 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d764d0b4ace4b6f90b529acb9ef8d72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:33:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37916
I20260925 16:33:03.957290 2553 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d764d0b4ace4b6f90b529acb9ef8d72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:33:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41490
I20260925 16:33:03.957670 2792 tablet_replica.cc:330] T 3d764d0b4ace4b6f90b529acb9ef8d72 P 435b9fb0fe93430face8f32e4de081b4: stopping tablet replica
I20260925 16:33:03.957686 2790 tablet_replica.cc:330] T 3d764d0b4ace4b6f90b529acb9ef8d72 P a3d3aa90f8e74bb2ae128513c9f0523a: stopping tablet replica
I20260925 16:33:03.957727 2791 tablet_replica.cc:330] T 3d764d0b4ace4b6f90b529acb9ef8d72 P 97509c025dbe428fabba21071efa72ce: stopping tablet replica
I20260925 16:33:03.957770 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 2208
I20260925 16:33:03.957876 2792 raft_consensus.cc:2229] T 3d764d0b4ace4b6f90b529acb9ef8d72 P 435b9fb0fe93430face8f32e4de081b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:33:03.957983 2792 raft_consensus.cc:2258] T 3d764d0b4ace4b6f90b529acb9ef8d72 P 435b9fb0fe93430face8f32e4de081b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:33:03.958603 2791 raft_consensus.cc:2229] T 3d764d0b4ace4b6f90b529acb9ef8d72 P 97509c025dbe428fabba21071efa72ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:33:03.958742 2791 raft_consensus.cc:2258] T 3d764d0b4ace4b6f90b529acb9ef8d72 P 97509c025dbe428fabba21071efa72ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:33:03.958827 2790 raft_consensus.cc:2229] T 3d764d0b4ace4b6f90b529acb9ef8d72 P a3d3aa90f8e74bb2ae128513c9f0523a [term 1 LEADER]: Raft consensus shutting down.
I20260925 16:33:03.959097 2790 raft_consensus.cc:2258] T 3d764d0b4ace4b6f90b529acb9ef8d72 P a3d3aa90f8e74bb2ae128513c9f0523a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:33:03.959515 2791 ts_tablet_manager.cc:1905] T 3d764d0b4ace4b6f90b529acb9ef8d72 P 97509c025dbe428fabba21071efa72ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:33:03.959893 2790 ts_tablet_manager.cc:1905] T 3d764d0b4ace4b6f90b529acb9ef8d72 P a3d3aa90f8e74bb2ae128513c9f0523a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:33:03.962667 2790 ts_tablet_manager.cc:1918] T 3d764d0b4ace4b6f90b529acb9ef8d72 P a3d3aa90f8e74bb2ae128513c9f0523a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 16:33:03.962770 2790 log.cc:1196] T 3d764d0b4ace4b6f90b529acb9ef8d72 P a3d3aa90f8e74bb2ae128513c9f0523a: Deleting WAL directory at /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/ts-2/wal/wals/3d764d0b4ace4b6f90b529acb9ef8d72
I20260925 16:33:03.962937 2791 ts_tablet_manager.cc:1918] T 3d764d0b4ace4b6f90b529acb9ef8d72 P 97509c025dbe428fabba21071efa72ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 16:33:03.963028 2791 log.cc:1196] T 3d764d0b4ace4b6f90b529acb9ef8d72 P 97509c025dbe428fabba21071efa72ce: Deleting WAL directory at /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353966814071-392-0/minicluster-data/ts-1/wal/wals/3d764d0b4ace4b6f90b529acb9ef8d72
I20260925 16:33:03.963202 2790 ts_tablet_manager.cc:1939] T 3d764d0b4ace4b6f90b529acb9ef8d72 P a3d3aa90f8e74bb2ae128513c9f0523a: Deleting consensus metadata
I20260925 16:33:03.963356 2791 ts_tablet_manager.cc:1939] T 3d764d0b4ace4b6f90b529acb9ef8d72 P 97509c025dbe428fabba21071efa72ce: Deleting consensus metadata
I20260925 16:33:03.964095 2122 catalog_manager.cc:4841] TS a3d3aa90f8e74bb2ae128513c9f0523a (127.0.98.3:43817): tablet 3d764d0b4ace4b6f90b529acb9ef8d72 (table test-table [id=44c2fe7773d146839f003f618fad18a8]) successfully deleted
I20260925 16:33:03.964151 2123 catalog_manager.cc:4841] TS 97509c025dbe428fabba21071efa72ce (127.0.98.2:37461): tablet 3d764d0b4ace4b6f90b529acb9ef8d72 (table test-table [id=44c2fe7773d146839f003f618fad18a8]) successfully deleted
W20260925 16:33:03.967689 2121 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43805 (error 108)
I20260925 16:33:03.967803 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 2348
I20260925 16:33:03.970220 2794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:03.969684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43805 (local address 127.0.0.1:58416)
0925 16:33:03.969977 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:03.970068 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":84,"threads_started":1}
W20260925 16:33:03.970376 2121 catalog_manager.cc:4584] TS 435b9fb0fe93430face8f32e4de081b4 (127.0.98.1:43805): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3d764d0b4ace4b6f90b529acb9ef8d72: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43805: connect: Connection refused (error 111)
I20260925 16:33:03.974956 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 2488
I20260925 16:33:03.981344 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 2104
2026-09-25T16:33:03Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6597 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:33:04 dist-test-slave-65pf krb5kdc[2797](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:33:04 dist-test-slave-65pf krb5kdc[2797](info): set up 2 sockets
Sep 25 16:33:04 dist-test-slave-65pf krb5kdc[2797](info): commencing operation
krb5kdc: starting...
W20260925 16:33:05.030471 392 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/krb5kdc/test-user.keytab.
Sep 25 16:33:05 dist-test-slave-65pf krb5kdc[2797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:33:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:33:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 16:33:05.149806 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38895
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38895
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:33:05.257515 2810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:33:05.257776 2810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:33:05.257817 2810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:33:05.261770 2810 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:33:05.261827 2810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:33:05.261847 2810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:33:05.261868 2810 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:33:05.261883 2810 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:33:05.267007 2810 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38895
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:33:05.268097 2810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:33:05.269100 2810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:33:05.277678 2816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:05.277673 2815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:05.277673 2818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:05.277757 2810 server_base.cc:1034] running on GCE node
I20260925 16:33:05.278563 2810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:33:05.279700 2810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:33:05.280857 2810 hybrid_clock.cc:648] HybridClock initialized: now 1790353985280841 us; error 29 us; skew 500 ppm
Sep 25 16:33:05 dist-test-slave-65pf krb5kdc[2797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:33:05.283703 2810 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 16:33:05.284590 2810 webserver.cc:460] Webserver started at http://127.0.98.62:32931/ using document root <none> and password file <none>
I20260925 16:33:05.285162 2810 fs_manager.cc:359] Metadata directory not provided
I20260925 16:33:05.285247 2810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:33:05.285468 2810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:33:05.287602 2810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/minicluster-data/master-0/data/instance:
uuid: "58842a4fda2f4f3a90a9292854fcf403"
format_stamp: "Formatted at 2026-09-25 16:33:05 on dist-test-slave-65pf"
server_key: "2071053157fde5e03d1dbcff8d3fab94"
server_key_iv: "5d5d97a9f90c614328faa586537a49f9"
server_key_version: "encryptionkey@0"
I20260925 16:33:05.288177 2810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/minicluster-data/master-0/wal/instance:
uuid: "58842a4fda2f4f3a90a9292854fcf403"
format_stamp: "Formatted at 2026-09-25 16:33:05 on dist-test-slave-65pf"
server_key: "2071053157fde5e03d1dbcff8d3fab94"
server_key_iv: "5d5d97a9f90c614328faa586537a49f9"
server_key_version: "encryptionkey@0"
I20260925 16:33:05.291952 2810 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 16:33:05.294255 2825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:05.295480 2810 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:33:05.295614 2810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/minicluster-data/master-0/wal
uuid: "58842a4fda2f4f3a90a9292854fcf403"
format_stamp: "Formatted at 2026-09-25 16:33:05 on dist-test-slave-65pf"
server_key: "2071053157fde5e03d1dbcff8d3fab94"
server_key_iv: "5d5d97a9f90c614328faa586537a49f9"
server_key_version: "encryptionkey@0"
I20260925 16:33:05.295719 2810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:33:05.321657 2810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:33:05.324769 2810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:33:05.324975 2810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:33:05.331797 2810 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38895
I20260925 16:33:05.331825 2877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38895 every 8 connection(s)
I20260925 16:33:05.332736 2810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:33:05.335393 2878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:33:05.335824 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 2810
I20260925 16:33:05.335924 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/minicluster-data/master-0/wal/instance
I20260925 16:33:05.336185 392 external_mini_cluster.cc:1428] Setting key 0a5b2f1b7dd7cfca173796d5a71581be
I20260925 16:33:05.342461 2878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58842a4fda2f4f3a90a9292854fcf403: Bootstrap starting.
I20260925 16:33:05.344653 2878 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58842a4fda2f4f3a90a9292854fcf403: Neither blocks nor log segments found. Creating new log.
Sep 25 16:33:05 dist-test-slave-65pf krb5kdc[2797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:33:05.345544 2878 log.cc:824] T 00000000000000000000000000000000 P 58842a4fda2f4f3a90a9292854fcf403: Log is configured to *not* fsync() on all Append() calls
I20260925 16:33:05.347815 2878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58842a4fda2f4f3a90a9292854fcf403: No bootstrap required, opened a new log
I20260925 16:33:05.350939 2881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:05.337532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35924 (local address 127.0.98.62:38895)
0925 16:33:05.337954 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:05.337973 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:05.338017 (+ 44us) server_negotiation.cc:407] Connection header received
0925 16:33:05.338827 (+ 810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:05.338851 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:05.339263 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:05.339637 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:05.340291 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:05.341774 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:05.342651 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:05.343123 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:05.343298 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:05.345931 (+ 2633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:05.345937 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:05.345954 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:05.345981 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:05.347878 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:05.348279 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:05.348284 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:05.348290 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:05.348443 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:05.348698 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:05.348701 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:05.348702 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:05.349176 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:05.349351 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:05.349554 (+ 203us) server_negotiation.cc:299] Negotiation successful
0925 16:33:05.349847 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":123,"threads_started":1}
I20260925 16:33:05.352283 2878 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58842a4fda2f4f3a90a9292854fcf403 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58842a4fda2f4f3a90a9292854fcf403" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38895 } }
I20260925 16:33:05.352530 2878 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58842a4fda2f4f3a90a9292854fcf403 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:33:05.352587 2878 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58842a4fda2f4f3a90a9292854fcf403 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58842a4fda2f4f3a90a9292854fcf403, State: Initialized, Role: FOLLOWER
I20260925 16:33:05.353066 2878 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58842a4fda2f4f3a90a9292854fcf403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58842a4fda2f4f3a90a9292854fcf403" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38895 } }
I20260925 16:33:05.353220 2878 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58842a4fda2f4f3a90a9292854fcf403 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:33:05.353286 2878 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58842a4fda2f4f3a90a9292854fcf403 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:33:05.353355 2878 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58842a4fda2f4f3a90a9292854fcf403 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:33:05.354252 2878 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58842a4fda2f4f3a90a9292854fcf403 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58842a4fda2f4f3a90a9292854fcf403" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38895 } }
I20260925 16:33:05.354528 2878 leader_election.cc:302] T 00000000000000000000000000000000 P 58842a4fda2f4f3a90a9292854fcf403 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58842a4fda2f4f3a90a9292854fcf403; no voters:
I20260925 16:33:05.354828 2878 leader_election.cc:288] T 00000000000000000000000000000000 P 58842a4fda2f4f3a90a9292854fcf403 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:33:05.354943 2883 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58842a4fda2f4f3a90a9292854fcf403 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:33:05.355182 2883 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58842a4fda2f4f3a90a9292854fcf403 [term 1 LEADER]: Becoming Leader. State: Replica: 58842a4fda2f4f3a90a9292854fcf403, State: Running, Role: LEADER
I20260925 16:33:05.355525 2883 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58842a4fda2f4f3a90a9292854fcf403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58842a4fda2f4f3a90a9292854fcf403" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38895 } }
I20260925 16:33:05.356025 2878 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58842a4fda2f4f3a90a9292854fcf403 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:33:05.357098 2884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58842a4fda2f4f3a90a9292854fcf403 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58842a4fda2f4f3a90a9292854fcf403" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58842a4fda2f4f3a90a9292854fcf403" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38895 } } }
I20260925 16:33:05.357245 2884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58842a4fda2f4f3a90a9292854fcf403 [sys.catalog]: This master's current role is: LEADER
I20260925 16:33:05.357672 2892 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:33:05.357882 2885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58842a4fda2f4f3a90a9292854fcf403 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58842a4fda2f4f3a90a9292854fcf403. Latest consensus state: current_term: 1 leader_uuid: "58842a4fda2f4f3a90a9292854fcf403" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58842a4fda2f4f3a90a9292854fcf403" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38895 } } }
I20260925 16:33:05.357997 2885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58842a4fda2f4f3a90a9292854fcf403 [sys.catalog]: This master's current role is: LEADER
I20260925 16:33:05.360571 2892 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:33:05.365895 2892 catalog_manager.cc:1347] Generated new cluster ID: 66b1c9a703b846198c0cbc77d3063c6f
I20260925 16:33:05.365957 2892 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:33:05.375339 2892 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:33:05.376534 2892 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:33:05.389169 2892 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58842a4fda2f4f3a90a9292854fcf403: Generated new TSK 0
I20260925 16:33:05.390038 2892 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 16:33:05.421710 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38895
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:33:05.534572 2904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:33:05.534900 2904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:33:05.534942 2904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:33:05.539067 2904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:33:05.539125 2904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:33:05.539181 2904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 16:33:05.544286 2904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2904
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:33:05.545265 2904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:33:05.546216 2904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:33:05.555824 2912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:05.555860 2909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:05.555964 2904 server_base.cc:1034] running on GCE node
W20260925 16:33:05.556149 2910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:05.556658 2904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:33:05.557322 2904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:33:05.558480 2904 hybrid_clock.cc:648] HybridClock initialized: now 1790353985558454 us; error 57 us; skew 500 ppm
Sep 25 16:33:05 dist-test-slave-65pf krb5kdc[2797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:33:05.561278 2904 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 16:33:05.562150 2904 webserver.cc:460] Webserver started at http://127.0.98.1:37723/ using document root <none> and password file <none>
I20260925 16:33:05.562772 2904 fs_manager.cc:359] Metadata directory not provided
I20260925 16:33:05.562861 2904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:33:05.563124 2904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:33:05.565187 2904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-0/data/instance:
uuid: "49a336ce98f044b0b06f7930f2f0b1ad"
format_stamp: "Formatted at 2026-09-25 16:33:05 on dist-test-slave-65pf"
server_key: "3913d80839a79aea91970631f217d792"
server_key_iv: "77f563f5c8c3b669b77c40e3d62fcd6e"
server_key_version: "encryptionkey@0"
I20260925 16:33:05.565788 2904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-0/wal/instance:
uuid: "49a336ce98f044b0b06f7930f2f0b1ad"
format_stamp: "Formatted at 2026-09-25 16:33:05 on dist-test-slave-65pf"
server_key: "3913d80839a79aea91970631f217d792"
server_key_iv: "77f563f5c8c3b669b77c40e3d62fcd6e"
server_key_version: "encryptionkey@0"
I20260925 16:33:05.569547 2904 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260925 16:33:05.571982 2919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:05.573163 2904 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 16:33:05.573302 2904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-0/wal
uuid: "49a336ce98f044b0b06f7930f2f0b1ad"
format_stamp: "Formatted at 2026-09-25 16:33:05 on dist-test-slave-65pf"
server_key: "3913d80839a79aea91970631f217d792"
server_key_iv: "77f563f5c8c3b669b77c40e3d62fcd6e"
server_key_version: "encryptionkey@0"
I20260925 16:33:05.573410 2904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:33:05.586362 2904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:33:05.589908 2904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:33:05.590130 2904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:33:05.590816 2904 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:33:05.591652 2904 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:33:05.591724 2904 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:05.591797 2904 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:33:05.591840 2904 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:05.601168 2904 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46841
I20260925 16:33:05.601191 3032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46841 every 8 connection(s)
I20260925 16:33:05.602099 2904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
I20260925 16:33:05.607657 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 2904
I20260925 16:33:05.607785 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-0/wal/instance
I20260925 16:33:05.608058 392 external_mini_cluster.cc:1428] Setting key 1339f222138db0c0bbbd2c1bd83dfdb8
Sep 25 16:33:05 dist-test-slave-65pf krb5kdc[2797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:33:05.616084 2881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:05.603888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52347 (local address 127.0.98.62:38895)
0925 16:33:05.604056 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:05.604060 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:05.604945 (+ 885us) server_negotiation.cc:407] Connection header received
0925 16:33:05.606335 (+ 1390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:05.606338 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:05.606394 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:05.606474 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:05.607651 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:05.608318 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:05.609678 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:05.609919 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:05.610029 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:05.612567 (+ 2538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:05.612571 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:05.612573 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:05.612596 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:05.614027 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:05.614530 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:05.614535 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:05.614537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:05.614681 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:05.615028 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:05.615034 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:05.615035 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:05.615214 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:05.615320 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:05.615790 (+ 470us) server_negotiation.cc:299] Negotiation successful
0925 16:33:05.615923 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 16:33:05.616900 3035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:05.604282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38895 (local address 127.0.98.1:52347)
0925 16:33:05.604790 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:05.604846 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:05.606124 (+ 1278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:05.606675 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:05.606682 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:05.607027 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:05.607498 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:05.607511 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:05.608443 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:05.608447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:05.609560 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:05.609571 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:05.610071 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:05.610075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:05.610243 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:05.610944 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:05.610966 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:05.612391 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:33:05.614143 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:05.614149 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:05.614164 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:05.614426 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:05.614777 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:05.614780 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:05.614782 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:05.614931 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:05.615316 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:33:05.615323 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:33:05.615567 (+ 244us) client_negotiation.cc:769] Sending connection context
0925 16:33:05.615752 (+ 185us) client_negotiation.cc:240] Negotiation successful
0925 16:33:05.616104 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":117,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 16:33:05.618095 3033 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38895
I20260925 16:33:05.618383 3033 heartbeater.cc:464] Registering TS with master...
I20260925 16:33:05.618970 3033 heartbeater.cc:511] Master 127.0.98.62:38895 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260925 16:33:05.620739 2842 ts_manager.cc:194] Registered new tserver with Master: 49a336ce98f044b0b06f7930f2f0b1ad (127.0.98.1:46841)
I20260925 16:33:05.622085 2842 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52347
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 16:33:05.635601 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38895
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:33:05.744971 3038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:33:05.745234 3038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:33:05.745275 3038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:33:05.749328 3038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:33:05.749390 3038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:33:05.749447 3038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 16:33:05.754771 3038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:33:05.755766 3038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:33:05.756829 3038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:33:05.766639 3044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:05.766613 3043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:05.766780 3038 server_base.cc:1034] running on GCE node
W20260925 16:33:05.766734 3046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:05.767331 3038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:33:05.767976 3038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:33:05.769168 3038 hybrid_clock.cc:648] HybridClock initialized: now 1790353985769152 us; error 31 us; skew 500 ppm
Sep 25 16:33:05 dist-test-slave-65pf krb5kdc[2797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:33:05.772818 3038 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 16:33:05.773731 3038 webserver.cc:460] Webserver started at http://127.0.98.2:38463/ using document root <none> and password file <none>
I20260925 16:33:05.774318 3038 fs_manager.cc:359] Metadata directory not provided
I20260925 16:33:05.774380 3038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:33:05.774665 3038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:33:05.776775 3038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-1/data/instance:
uuid: "8364645922ea4346b86869c7e9e7c08b"
format_stamp: "Formatted at 2026-09-25 16:33:05 on dist-test-slave-65pf"
server_key: "35b8448f8142fa77c08d19b7fc2b9f02"
server_key_iv: "6cae50418b67c6488ac1f9bcf88981bf"
server_key_version: "encryptionkey@0"
I20260925 16:33:05.777352 3038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8364645922ea4346b86869c7e9e7c08b"
format_stamp: "Formatted at 2026-09-25 16:33:05 on dist-test-slave-65pf"
server_key: "35b8448f8142fa77c08d19b7fc2b9f02"
server_key_iv: "6cae50418b67c6488ac1f9bcf88981bf"
server_key_version: "encryptionkey@0"
I20260925 16:33:05.781033 3038 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 16:33:05.783519 3053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:05.784772 3038 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:33:05.784865 3038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-1/wal
uuid: "8364645922ea4346b86869c7e9e7c08b"
format_stamp: "Formatted at 2026-09-25 16:33:05 on dist-test-slave-65pf"
server_key: "35b8448f8142fa77c08d19b7fc2b9f02"
server_key_iv: "6cae50418b67c6488ac1f9bcf88981bf"
server_key_version: "encryptionkey@0"
I20260925 16:33:05.785004 3038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:33:05.801806 3038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:33:05.805058 3038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:33:05.805253 3038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:33:05.805822 3038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:33:05.806684 3038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:33:05.806756 3038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:05.806826 3038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:33:05.806885 3038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:05.815842 3038 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45479
I20260925 16:33:05.815866 3166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45479 every 8 connection(s)
I20260925 16:33:05.816834 3038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
I20260925 16:33:05.821782 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 3038
I20260925 16:33:05.821907 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-1/wal/instance
I20260925 16:33:05.822208 392 external_mini_cluster.cc:1428] Setting key 1f926ea5ab68d05deaa7339dd601b528
Sep 25 16:33:05 dist-test-slave-65pf krb5kdc[2797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260925 16:33:05.832267 2881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:05.818732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48585 (local address 127.0.98.62:38895)
0925 16:33:05.818896 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:05.818902 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:05.819797 (+ 895us) server_negotiation.cc:407] Connection header received
0925 16:33:05.821131 (+ 1334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:05.821134 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:05.821179 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:05.821256 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:05.822834 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:05.823582 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:05.825310 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:05.825703 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:05.825821 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:05.828554 (+ 2733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:05.828558 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:05.828563 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:05.828592 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:05.830067 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:05.830630 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:05.830638 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:05.830640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:05.830716 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:05.831071 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:05.831076 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:05.831082 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:05.831262 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:05.831370 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:05.831952 (+ 582us) server_negotiation.cc:299] Negotiation successful
0925 16:33:05.832090 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 16:33:05.833077 3169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:05.819188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38895 (local address 127.0.98.2:48585)
0925 16:33:05.819636 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:05.819677 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:05.820902 (+ 1225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:05.821422 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:05.821429 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:05.821975 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:05.822674 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:05.822692 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:05.823703 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:05.823706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:05.825167 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:05.825181 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:05.825868 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:05.825871 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:05.826013 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:05.826679 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:05.826696 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:05.828377 (+ 1681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:33:05.830194 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:05.830201 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:05.830217 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:05.830484 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:05.830809 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:05.830813 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:05.830815 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:05.830962 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:05.831369 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:33:05.831378 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:33:05.831650 (+ 272us) client_negotiation.cc:769] Sending connection context
0925 16:33:05.831891 (+ 241us) client_negotiation.cc:240] Negotiation successful
0925 16:33:05.832233 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":115,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260925 16:33:05.834223 3167 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38895
I20260925 16:33:05.834482 3167 heartbeater.cc:464] Registering TS with master...
I20260925 16:33:05.835134 3167 heartbeater.cc:511] Master 127.0.98.62:38895 requested a full tablet report, sending...
I20260925 16:33:05.836437 2842 ts_manager.cc:194] Registered new tserver with Master: 8364645922ea4346b86869c7e9e7c08b (127.0.98.2:45479)
I20260925 16:33:05.837245 2842 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48585
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 16:33:05.850334 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38895
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:33:05.962266 3172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:33:05.962626 3172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:33:05.962740 3172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:33:05.966843 3172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:33:05.966918 3172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:33:05.967024 3172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 16:33:05.972545 3172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:33:05.973660 3172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:33:05.974742 3172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:33:05.984481 3178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:05.984547 3172 server_base.cc:1034] running on GCE node
W20260925 16:33:05.984488 3177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:05.984481 3180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:05.985100 3172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:33:05.985736 3172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:33:05.986891 3172 hybrid_clock.cc:648] HybridClock initialized: now 1790353985986876 us; error 30 us; skew 500 ppm
Sep 25 16:33:05 dist-test-slave-65pf krb5kdc[2797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:33:05.989854 3172 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 16:33:05.990818 3172 webserver.cc:460] Webserver started at http://127.0.98.3:43281/ using document root <none> and password file <none>
I20260925 16:33:05.991380 3172 fs_manager.cc:359] Metadata directory not provided
I20260925 16:33:05.991474 3172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:33:05.991714 3172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:33:05.993757 3172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-2/data/instance:
uuid: "ca5132a8131f43d5b1e03ab0d6f82999"
format_stamp: "Formatted at 2026-09-25 16:33:05 on dist-test-slave-65pf"
server_key: "f3cb8abcd4614b87f7c0b608dc759b7a"
server_key_iv: "34bb9cbc16e00207c03c31105ad67bf3"
server_key_version: "encryptionkey@0"
I20260925 16:33:05.994333 3172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ca5132a8131f43d5b1e03ab0d6f82999"
format_stamp: "Formatted at 2026-09-25 16:33:05 on dist-test-slave-65pf"
server_key: "f3cb8abcd4614b87f7c0b608dc759b7a"
server_key_iv: "34bb9cbc16e00207c03c31105ad67bf3"
server_key_version: "encryptionkey@0"
I20260925 16:33:05.998142 3172 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 16:33:06.000859 3187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:06.002074 3172 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:33:06.002210 3172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-2/wal
uuid: "ca5132a8131f43d5b1e03ab0d6f82999"
format_stamp: "Formatted at 2026-09-25 16:33:05 on dist-test-slave-65pf"
server_key: "f3cb8abcd4614b87f7c0b608dc759b7a"
server_key_iv: "34bb9cbc16e00207c03c31105ad67bf3"
server_key_version: "encryptionkey@0"
I20260925 16:33:06.002326 3172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:33:06.019497 3172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:33:06.022317 3172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:33:06.022531 3172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:33:06.023172 3172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:33:06.024035 3172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:33:06.024107 3172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:06.024181 3172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:33:06.024230 3172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:06.032833 3172 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44079
I20260925 16:33:06.032871 3300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44079 every 8 connection(s)
I20260925 16:33:06.033721 3172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
I20260925 16:33:06.036631 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 3172
I20260925 16:33:06.036778 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-2/wal/instance
I20260925 16:33:06.037103 392 external_mini_cluster.cc:1428] Setting key d9e1a096fe4b61adddea9c22f65fb150
Sep 25 16:33:06 dist-test-slave-65pf krb5kdc[2797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:33:06.048516 2881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:06.035640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47153 (local address 127.0.98.62:38895)
0925 16:33:06.035798 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:06.035803 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:06.036558 (+ 755us) server_negotiation.cc:407] Connection header received
0925 16:33:06.038509 (+ 1951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:06.038512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:06.038568 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:06.038673 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:06.040156 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:06.040672 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:06.041972 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:06.042224 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:06.042310 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:06.045205 (+ 2895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:06.045208 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:06.045210 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:06.045232 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:06.046706 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:06.047181 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:06.047183 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:06.047184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:06.047254 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:06.047542 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:06.047544 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:06.047545 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:06.047719 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:06.047795 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:06.048287 (+ 492us) server_negotiation.cc:299] Negotiation successful
0925 16:33:06.048383 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 16:33:06.049348 3303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:06.035895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38895 (local address 127.0.98.3:47153)
0925 16:33:06.036390 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:06.036437 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:06.038234 (+ 1797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:06.038876 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:06.038888 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:06.039468 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:06.040018 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:06.040033 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:06.040803 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:06.040807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:06.041842 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:06.041852 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:06.042392 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:06.042396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:06.042554 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:06.043380 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:06.043399 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:06.045041 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:33:06.046806 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:06.046811 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:06.046824 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:06.047091 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:06.047332 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:06.047334 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:06.047335 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:06.047466 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:06.047804 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:33:06.047808 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:33:06.048085 (+ 277us) client_negotiation.cc:769] Sending connection context
0925 16:33:06.048244 (+ 159us) client_negotiation.cc:240] Negotiation successful
0925 16:33:06.048506 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":111,"threads_started":1}
I20260925 16:33:06.050722 3301 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38895
I20260925 16:33:06.051005 3301 heartbeater.cc:464] Registering TS with master...
I20260925 16:33:06.051573 3301 heartbeater.cc:511] Master 127.0.98.62:38895 requested a full tablet report, sending...
I20260925 16:33:06.052518 2841 ts_manager.cc:194] Registered new tserver with Master: ca5132a8131f43d5b1e03ab0d6f82999 (127.0.98.3:44079)
I20260925 16:33:06.053156 2841 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47153
I20260925 16:33:06.062989 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 16:33:06.075763 2881 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:33:06.074276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40094 (local address 127.0.98.62:38895)
0925 16:33:06.074405 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:06.074409 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:06.074634 (+ 225us) server_negotiation.cc:407] Connection header received
0925 16:33:06.074767 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:06.074770 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:06.074814 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:06.074932 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:06.075601 (+ 669us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40094: BlockingRecv error: recv got EOF from 127.0.0.1:40094 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 16:33:06.077095 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 2904
I20260925 16:33:06.083055 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 3038
I20260925 16:33:06.088714 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 3172
I20260925 16:33:06.094302 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 2810
2026-09-25T16:33:06Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2110 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790353966814071-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:33:06 dist-test-slave-65pf krb5kdc[3313](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:33:06 dist-test-slave-65pf krb5kdc[3313](info): set up 2 sockets
Sep 25 16:33:06 dist-test-slave-65pf krb5kdc[3313](info): commencing operation
krb5kdc: starting...
W20260925 16:33:07.138528 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790353966814071-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790353966814071-392-0/krb5kdc/test-user.keytab.
Sep 25 16:33:07 dist-test-slave-65pf krb5kdc[3313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:33:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:33:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 16:33:07.257728 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790353966814071-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790353966814071-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790353966814071-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46535
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46535
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:33:07.365795 3326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:33:07.366070 3326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:33:07.366111 3326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:33:07.370133 3326 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:33:07.370198 3326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:33:07.370217 3326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:33:07.370234 3326 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:33:07.370249 3326 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:33:07.375618 3326 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790353966814071-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790353966814071-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46535
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3326
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790353966814071-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:33:07.376710 3326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:33:07.377792 3326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:33:07.386370 3334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:07.386370 3332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:07.386397 3331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:07.386762 3326 server_base.cc:1034] running on GCE node
I20260925 16:33:07.387346 3326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:33:07.388479 3326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:33:07.389659 3326 hybrid_clock.cc:648] HybridClock initialized: now 1790353987389628 us; error 46 us; skew 500 ppm
Sep 25 16:33:07 dist-test-slave-65pf krb5kdc[3313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353987, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:33:07.396183 3326 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 16:33:07.397126 3326 webserver.cc:460] Webserver started at http://127.0.98.62:42947/ using document root <none> and password file <none>
I20260925 16:33:07.397696 3326 fs_manager.cc:359] Metadata directory not provided
I20260925 16:33:07.397785 3326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:33:07.398015 3326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:33:07.400151 3326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790353966814071-392-0/minicluster-data/master-0/data/instance:
uuid: "6f3b76ccf1d5410283d670ef42de8009"
format_stamp: "Formatted at 2026-09-25 16:33:07 on dist-test-slave-65pf"
server_key: "9c347a777e9e7ab56c2664c85190ea5d"
server_key_iv: "1450e97f465b20d730edb5200c40885e"
server_key_version: "encryptionkey@0"
I20260925 16:33:07.400736 3326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790353966814071-392-0/minicluster-data/master-0/wal/instance:
uuid: "6f3b76ccf1d5410283d670ef42de8009"
format_stamp: "Formatted at 2026-09-25 16:33:07 on dist-test-slave-65pf"
server_key: "9c347a777e9e7ab56c2664c85190ea5d"
server_key_iv: "1450e97f465b20d730edb5200c40885e"
server_key_version: "encryptionkey@0"
I20260925 16:33:07.404556 3326 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260925 16:33:07.406914 3341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:07.408159 3326 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 16:33:07.408298 3326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790353966814071-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790353966814071-392-0/minicluster-data/master-0/wal
uuid: "6f3b76ccf1d5410283d670ef42de8009"
format_stamp: "Formatted at 2026-09-25 16:33:07 on dist-test-slave-65pf"
server_key: "9c347a777e9e7ab56c2664c85190ea5d"
server_key_iv: "1450e97f465b20d730edb5200c40885e"
server_key_version: "encryptionkey@0"
I20260925 16:33:07.408418 3326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790353966814071-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790353966814071-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790353966814071-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:33:07.421264 3326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:33:07.424155 3326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:33:07.424360 3326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:33:07.431049 3326 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46535
I20260925 16:33:07.431334 3393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46535 every 8 connection(s)
I20260925 16:33:07.431986 3326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:33:07.433763 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 3326
I20260925 16:33:07.433913 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790353966814071-392-0/minicluster-data/master-0/wal/instance
I20260925 16:33:07.434190 392 external_mini_cluster.cc:1428] Setting key b61e505d54b4509f460c4ee27bbac077
I20260925 16:33:07.434937 3394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:33:07.441519 3394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f3b76ccf1d5410283d670ef42de8009: Bootstrap starting.
Sep 25 16:33:07 dist-test-slave-65pf krb5kdc[3313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:33:07.444335 3394 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f3b76ccf1d5410283d670ef42de8009: Neither blocks nor log segments found. Creating new log.
I20260925 16:33:07.445247 3394 log.cc:824] T 00000000000000000000000000000000 P 6f3b76ccf1d5410283d670ef42de8009: Log is configured to *not* fsync() on all Append() calls
I20260925 16:33:07.447615 3394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f3b76ccf1d5410283d670ef42de8009: No bootstrap required, opened a new log
I20260925 16:33:07.448963 3397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:07.435461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51448 (local address 127.0.98.62:46535)
0925 16:33:07.435869 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:07.435883 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:07.435920 (+ 37us) server_negotiation.cc:407] Connection header received
0925 16:33:07.436544 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:07.436562 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:07.436874 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:07.437147 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:07.438120 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:07.439223 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:07.440461 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:07.440939 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:07.441070 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:07.443805 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:07.443811 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:07.443827 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:07.443856 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:07.445868 (+ 2012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:07.446321 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:07.446327 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:07.446333 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:07.446458 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:07.446761 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:07.446767 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:07.446771 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:07.447270 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:07.447465 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:07.447749 (+ 284us) server_negotiation.cc:299] Negotiation successful
0925 16:33:07.448062 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":143,"threads_started":1}
I20260925 16:33:07.451993 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:33:07.452030 3394 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f3b76ccf1d5410283d670ef42de8009 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f3b76ccf1d5410283d670ef42de8009" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46535 } }
I20260925 16:33:07.452294 3394 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f3b76ccf1d5410283d670ef42de8009 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:33:07.452365 3394 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f3b76ccf1d5410283d670ef42de8009 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f3b76ccf1d5410283d670ef42de8009, State: Initialized, Role: FOLLOWER
I20260925 16:33:07.452834 3394 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f3b76ccf1d5410283d670ef42de8009 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f3b76ccf1d5410283d670ef42de8009" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46535 } }
I20260925 16:33:07.452960 3394 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f3b76ccf1d5410283d670ef42de8009 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:33:07.453045 3394 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f3b76ccf1d5410283d670ef42de8009 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:33:07.453155 3394 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f3b76ccf1d5410283d670ef42de8009 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:33:07.454048 3394 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f3b76ccf1d5410283d670ef42de8009 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f3b76ccf1d5410283d670ef42de8009" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46535 } }
I20260925 16:33:07.454391 3394 leader_election.cc:302] T 00000000000000000000000000000000 P 6f3b76ccf1d5410283d670ef42de8009 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f3b76ccf1d5410283d670ef42de8009; no voters:
I20260925 16:33:07.454751 3394 leader_election.cc:288] T 00000000000000000000000000000000 P 6f3b76ccf1d5410283d670ef42de8009 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:33:07.454879 3400 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f3b76ccf1d5410283d670ef42de8009 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:33:07.455124 3400 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f3b76ccf1d5410283d670ef42de8009 [term 1 LEADER]: Becoming Leader. State: Replica: 6f3b76ccf1d5410283d670ef42de8009, State: Running, Role: LEADER
I20260925 16:33:07.455521 3400 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f3b76ccf1d5410283d670ef42de8009 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f3b76ccf1d5410283d670ef42de8009" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46535 } }
I20260925 16:33:07.456068 3394 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f3b76ccf1d5410283d670ef42de8009 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:33:07.457249 3402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f3b76ccf1d5410283d670ef42de8009 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f3b76ccf1d5410283d670ef42de8009" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f3b76ccf1d5410283d670ef42de8009" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46535 } } }
I20260925 16:33:07.457370 3402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f3b76ccf1d5410283d670ef42de8009 [sys.catalog]: This master's current role is: LEADER
I20260925 16:33:07.457593 3402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f3b76ccf1d5410283d670ef42de8009 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f3b76ccf1d5410283d670ef42de8009. Latest consensus state: current_term: 1 leader_uuid: "6f3b76ccf1d5410283d670ef42de8009" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f3b76ccf1d5410283d670ef42de8009" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46535 } } }
I20260925 16:33:07.457692 3402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f3b76ccf1d5410283d670ef42de8009 [sys.catalog]: This master's current role is: LEADER
I20260925 16:33:07.458135 3409 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:33:07.461299 3409 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:33:07.467940 3409 catalog_manager.cc:1347] Generated new cluster ID: fdd688ec78fc4c4c9746a32b398ecc70
I20260925 16:33:07.468007 3409 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:33:07.473585 3397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:07.467910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51458 (local address 127.0.98.62:46535)
0925 16:33:07.468044 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:07.468049 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:07.468332 (+ 283us) server_negotiation.cc:407] Connection header received
0925 16:33:07.468465 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:07.468468 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:07.468530 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:07.468630 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:07.469211 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:07.470032 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:07.471186 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:07.471561 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:07.471671 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:07.472565 (+ 894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:07.472569 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:07.472572 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:33:07.473106 (+ 534us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:07.473140 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:33:07.473159 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:07.473251 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:07.473394 (+ 143us) server_negotiation.cc:299] Negotiation successful
0925 16:33:07.473450 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 16:33:07.485863 3409 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:33:07.486914 3409 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:33:07.498140 3409 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f3b76ccf1d5410283d670ef42de8009: Generated new TSK 0
I20260925 16:33:07.498929 3409 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:33:07.509862 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 3326
2026-09-25T16:33:07Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1413 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:33:07 dist-test-slave-65pf krb5kdc[3427](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:33:07 dist-test-slave-65pf krb5kdc[3427](info): set up 2 sockets
Sep 25 16:33:07 dist-test-slave-65pf krb5kdc[3427](info): commencing operation
krb5kdc: starting...
W20260925 16:33:08.546710 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/krb5kdc/test-user.keytab.
Sep 25 16:33:08 dist-test-slave-65pf krb5kdc[3427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:33:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:33:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 16:33:08.665977 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35061
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35061
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:33:08.774717 3440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:33:08.775032 3440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:33:08.775117 3440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:33:08.779186 3440 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:33:08.779263 3440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:33:08.779300 3440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:33:08.779356 3440 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:33:08.779404 3440 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:33:08.785790 3440 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35061
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:33:08.786968 3440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:33:08.788002 3440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:33:08.796437 3445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:08.796460 3446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:08.796473 3448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:08.796599 3440 server_base.cc:1034] running on GCE node
I20260925 16:33:08.797197 3440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:33:08.798255 3440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:33:08.799444 3440 hybrid_clock.cc:648] HybridClock initialized: now 1790353988799437 us; error 32 us; skew 500 ppm
Sep 25 16:33:08 dist-test-slave-65pf krb5kdc[3427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353988, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:33:08.802126 3440 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 16:33:08.803043 3440 webserver.cc:460] Webserver started at http://127.0.98.62:37195/ using document root <none> and password file <none>
I20260925 16:33:08.803541 3440 fs_manager.cc:359] Metadata directory not provided
I20260925 16:33:08.803598 3440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:33:08.803794 3440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:33:08.805787 3440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/minicluster-data/master-0/data/instance:
uuid: "58942a1c78044fb48eaac4bd8a06a5e5"
format_stamp: "Formatted at 2026-09-25 16:33:08 on dist-test-slave-65pf"
server_key: "8004af4d0cc4845e333cf95c9aedaf26"
server_key_iv: "b082916181286ba30c09cd9f0db83ae5"
server_key_version: "encryptionkey@0"
I20260925 16:33:08.806291 3440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/minicluster-data/master-0/wal/instance:
uuid: "58942a1c78044fb48eaac4bd8a06a5e5"
format_stamp: "Formatted at 2026-09-25 16:33:08 on dist-test-slave-65pf"
server_key: "8004af4d0cc4845e333cf95c9aedaf26"
server_key_iv: "b082916181286ba30c09cd9f0db83ae5"
server_key_version: "encryptionkey@0"
I20260925 16:33:08.809916 3440 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 16:33:08.812269 3455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:08.813410 3440 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 16:33:08.813510 3440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/minicluster-data/master-0/wal
uuid: "58942a1c78044fb48eaac4bd8a06a5e5"
format_stamp: "Formatted at 2026-09-25 16:33:08 on dist-test-slave-65pf"
server_key: "8004af4d0cc4845e333cf95c9aedaf26"
server_key_iv: "b082916181286ba30c09cd9f0db83ae5"
server_key_version: "encryptionkey@0"
I20260925 16:33:08.813594 3440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:33:08.828656 3440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:33:08.831612 3440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:33:08.831779 3440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:33:08.838961 3440 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35061
I20260925 16:33:08.839298 3507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35061 every 8 connection(s)
I20260925 16:33:08.840018 3440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:33:08.842127 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 3440
I20260925 16:33:08.842247 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/minicluster-data/master-0/wal/instance
I20260925 16:33:08.842556 392 external_mini_cluster.cc:1428] Setting key aa2e856726eeae741916d376b0c7850c
I20260925 16:33:08.843088 3508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:33:08.849056 3508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58942a1c78044fb48eaac4bd8a06a5e5: Bootstrap starting.
Sep 25 16:33:08 dist-test-slave-65pf krb5kdc[3427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:33:08.851608 3508 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58942a1c78044fb48eaac4bd8a06a5e5: Neither blocks nor log segments found. Creating new log.
I20260925 16:33:08.852469 3508 log.cc:824] T 00000000000000000000000000000000 P 58942a1c78044fb48eaac4bd8a06a5e5: Log is configured to *not* fsync() on all Append() calls
I20260925 16:33:08.854655 3508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58942a1c78044fb48eaac4bd8a06a5e5: No bootstrap required, opened a new log
I20260925 16:33:08.856648 3511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:08.844061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35298 (local address 127.0.98.62:35061)
0925 16:33:08.844495 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:08.844509 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:08.844544 (+ 35us) server_negotiation.cc:407] Connection header received
0925 16:33:08.845235 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:08.845254 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:08.845580 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:08.845929 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:08.846769 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:08.847768 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:08.848630 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:08.849092 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:08.849282 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:08.852088 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:08.852093 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:08.852107 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:08.852133 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:08.853865 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:08.854381 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:08.854385 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:08.854389 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:08.854490 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:08.854801 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:08.854803 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:08.854804 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:08.855174 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:08.855317 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:08.855618 (+ 301us) server_negotiation.cc:299] Negotiation successful
0925 16:33:08.855915 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":136,"threads_started":1}
I20260925 16:33:08.858669 3508 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58942a1c78044fb48eaac4bd8a06a5e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58942a1c78044fb48eaac4bd8a06a5e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35061 } }
I20260925 16:33:08.858916 3508 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58942a1c78044fb48eaac4bd8a06a5e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:33:08.858990 3508 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58942a1c78044fb48eaac4bd8a06a5e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58942a1c78044fb48eaac4bd8a06a5e5, State: Initialized, Role: FOLLOWER
I20260925 16:33:08.859424 3508 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58942a1c78044fb48eaac4bd8a06a5e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58942a1c78044fb48eaac4bd8a06a5e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35061 } }
I20260925 16:33:08.859572 3508 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58942a1c78044fb48eaac4bd8a06a5e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:33:08.859642 3508 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58942a1c78044fb48eaac4bd8a06a5e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:33:08.859755 3508 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58942a1c78044fb48eaac4bd8a06a5e5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:33:08.860652 3508 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58942a1c78044fb48eaac4bd8a06a5e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58942a1c78044fb48eaac4bd8a06a5e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35061 } }
I20260925 16:33:08.860991 3508 leader_election.cc:302] T 00000000000000000000000000000000 P 58942a1c78044fb48eaac4bd8a06a5e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58942a1c78044fb48eaac4bd8a06a5e5; no voters:
I20260925 16:33:08.861308 3508 leader_election.cc:288] T 00000000000000000000000000000000 P 58942a1c78044fb48eaac4bd8a06a5e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:33:08.861385 3513 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58942a1c78044fb48eaac4bd8a06a5e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:33:08.861593 3513 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58942a1c78044fb48eaac4bd8a06a5e5 [term 1 LEADER]: Becoming Leader. State: Replica: 58942a1c78044fb48eaac4bd8a06a5e5, State: Running, Role: LEADER
I20260925 16:33:08.861925 3513 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58942a1c78044fb48eaac4bd8a06a5e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58942a1c78044fb48eaac4bd8a06a5e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35061 } }
I20260925 16:33:08.862646 3508 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58942a1c78044fb48eaac4bd8a06a5e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:33:08.863761 3514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58942a1c78044fb48eaac4bd8a06a5e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58942a1c78044fb48eaac4bd8a06a5e5. Latest consensus state: current_term: 1 leader_uuid: "58942a1c78044fb48eaac4bd8a06a5e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58942a1c78044fb48eaac4bd8a06a5e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35061 } } }
I20260925 16:33:08.863796 3515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58942a1c78044fb48eaac4bd8a06a5e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58942a1c78044fb48eaac4bd8a06a5e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58942a1c78044fb48eaac4bd8a06a5e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35061 } } }
I20260925 16:33:08.863996 3515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58942a1c78044fb48eaac4bd8a06a5e5 [sys.catalog]: This master's current role is: LEADER
I20260925 16:33:08.864408 3522 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:33:08.864398 3514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58942a1c78044fb48eaac4bd8a06a5e5 [sys.catalog]: This master's current role is: LEADER
I20260925 16:33:08.867532 3522 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:33:08.873479 3522 catalog_manager.cc:1347] Generated new cluster ID: 4a97c13999c34246982483cd7cb1daf7
I20260925 16:33:08.873558 3522 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:33:08.883301 3522 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:33:08.884502 3522 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:33:08.889667 3522 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58942a1c78044fb48eaac4bd8a06a5e5: Generated new TSK 0
I20260925 16:33:08.890355 3522 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 16:33:08.921675 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35061
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:33:09.033531 3534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:33:09.033802 3534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:33:09.033847 3534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:33:09.037945 3534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:33:09.037998 3534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:33:09.038053 3534 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 16:33:09.043300 3534 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3534
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:33:09.044269 3534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:33:09.045240 3534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:33:09.054823 3540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:09.054912 3534 server_base.cc:1034] running on GCE node
W20260925 16:33:09.054883 3542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:09.054823 3539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:09.055543 3534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:33:09.056216 3534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:33:09.057425 3534 hybrid_clock.cc:648] HybridClock initialized: now 1790353989057408 us; error 30 us; skew 500 ppm
Sep 25 16:33:09 dist-test-slave-65pf krb5kdc[3427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:33:09.060129 3534 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 16:33:09.061048 3534 webserver.cc:460] Webserver started at http://127.0.98.1:38981/ using document root <none> and password file <none>
I20260925 16:33:09.061614 3534 fs_manager.cc:359] Metadata directory not provided
I20260925 16:33:09.061707 3534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:33:09.061942 3534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:33:09.064145 3534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-0/data/instance:
uuid: "acda477e03b246569ec506c15c5137b2"
format_stamp: "Formatted at 2026-09-25 16:33:09 on dist-test-slave-65pf"
server_key: "5f7dfc85b02381f22e7c9f54d0666cfb"
server_key_iv: "3840075875f5c1d3215ee2aed8b1b150"
server_key_version: "encryptionkey@0"
I20260925 16:33:09.064710 3534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-0/wal/instance:
uuid: "acda477e03b246569ec506c15c5137b2"
format_stamp: "Formatted at 2026-09-25 16:33:09 on dist-test-slave-65pf"
server_key: "5f7dfc85b02381f22e7c9f54d0666cfb"
server_key_iv: "3840075875f5c1d3215ee2aed8b1b150"
server_key_version: "encryptionkey@0"
I20260925 16:33:09.068631 3534 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 16:33:09.071062 3549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:09.072291 3534 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:33:09.072429 3534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-0/wal
uuid: "acda477e03b246569ec506c15c5137b2"
format_stamp: "Formatted at 2026-09-25 16:33:09 on dist-test-slave-65pf"
server_key: "5f7dfc85b02381f22e7c9f54d0666cfb"
server_key_iv: "3840075875f5c1d3215ee2aed8b1b150"
server_key_version: "encryptionkey@0"
I20260925 16:33:09.072543 3534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:33:09.085834 3534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:33:09.088908 3534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:33:09.089118 3534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:33:09.089712 3534 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:33:09.090556 3534 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:33:09.090660 3534 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:09.090737 3534 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:33:09.090791 3534 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:09.099800 3534 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44201
I20260925 16:33:09.099823 3662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44201 every 8 connection(s)
I20260925 16:33:09.100728 3534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
I20260925 16:33:09.108112 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 3534
I20260925 16:33:09.108230 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-0/wal/instance
I20260925 16:33:09.108516 392 external_mini_cluster.cc:1428] Setting key 7557d6af9a09abd80456b57efa4c46d1
Sep 25 16:33:09 dist-test-slave-65pf krb5kdc[3427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:33:09.115614 3511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:09.102629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41547 (local address 127.0.98.62:35061)
0925 16:33:09.102752 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:09.102755 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:09.103666 (+ 911us) server_negotiation.cc:407] Connection header received
0925 16:33:09.105120 (+ 1454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:09.105123 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:09.105183 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:09.105295 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:09.106532 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:09.107145 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:09.108386 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:09.108662 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:09.108776 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:09.111707 (+ 2931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:09.111711 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:09.111714 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:09.111742 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:09.113280 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:09.113913 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:09.113916 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:09.113917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:09.113988 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:09.114384 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:09.114386 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:09.114387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:09.114556 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:09.114711 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:09.115382 (+ 671us) server_negotiation.cc:299] Negotiation successful
0925 16:33:09.115490 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 16:33:09.117015 3665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:09.103016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35061 (local address 127.0.98.1:41547)
0925 16:33:09.103498 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:09.103543 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:09.104886 (+ 1343us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:09.105492 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:09.105499 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:09.105883 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:09.106374 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:09.106388 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:09.107278 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:09.107281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:09.108169 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:09.108208 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:09.108831 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:09.108835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:09.109005 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:09.109587 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:09.109607 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:09.111517 (+ 1910us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:33:09.113392 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:09.113401 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:09.113421 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:09.113807 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:09.114079 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:09.114084 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:09.114089 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:09.114282 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:09.114706 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:33:09.114716 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:33:09.115100 (+ 384us) client_negotiation.cc:769] Sending connection context
0925 16:33:09.115343 (+ 243us) client_negotiation.cc:240] Negotiation successful
0925 16:33:09.115751 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":121,"threads_started":1}
I20260925 16:33:09.118456 3663 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35061
I20260925 16:33:09.118737 3663 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 16:33:09.119292 3663 heartbeater.cc:511] Master 127.0.98.62:35061 requested a full tablet report, sending...
I20260925 16:33:09.121088 3472 ts_manager.cc:194] Registered new tserver with Master: acda477e03b246569ec506c15c5137b2 (127.0.98.1:44201)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260925 16:33:09.123034 3472 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41547
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 16:33:09.137673 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35061
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:33:09.249356 3668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:33:09.249686 3668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:33:09.249761 3668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:33:09.254065 3668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:33:09.254161 3668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:33:09.254251 3668 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 16:33:09.259750 3668 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3668
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:33:09.260848 3668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:33:09.261921 3668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:33:09.272751 3673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:09.272779 3676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:09.272938 3668 server_base.cc:1034] running on GCE node
W20260925 16:33:09.272739 3674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:09.273581 3668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:33:09.274300 3668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:33:09.275494 3668 hybrid_clock.cc:648] HybridClock initialized: now 1790353989275432 us; error 77 us; skew 500 ppm
Sep 25 16:33:09 dist-test-slave-65pf krb5kdc[3427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:33:09.278329 3668 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 16:33:09.279301 3668 webserver.cc:460] Webserver started at http://127.0.98.2:43139/ using document root <none> and password file <none>
I20260925 16:33:09.279886 3668 fs_manager.cc:359] Metadata directory not provided
I20260925 16:33:09.279985 3668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:33:09.280221 3668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:33:09.282367 3668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-1/data/instance:
uuid: "2988884eefe14f41a27e1496b1bf6a0a"
format_stamp: "Formatted at 2026-09-25 16:33:09 on dist-test-slave-65pf"
server_key: "b750587f9e3923a48d440e0a97e226cc"
server_key_iv: "a854f2f61bfd40aed9d74fb9c894eeae"
server_key_version: "encryptionkey@0"
I20260925 16:33:09.282992 3668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2988884eefe14f41a27e1496b1bf6a0a"
format_stamp: "Formatted at 2026-09-25 16:33:09 on dist-test-slave-65pf"
server_key: "b750587f9e3923a48d440e0a97e226cc"
server_key_iv: "a854f2f61bfd40aed9d74fb9c894eeae"
server_key_version: "encryptionkey@0"
I20260925 16:33:09.286765 3668 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 16:33:09.289233 3683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:09.290464 3668 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:33:09.290620 3668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-1/wal
uuid: "2988884eefe14f41a27e1496b1bf6a0a"
format_stamp: "Formatted at 2026-09-25 16:33:09 on dist-test-slave-65pf"
server_key: "b750587f9e3923a48d440e0a97e226cc"
server_key_iv: "a854f2f61bfd40aed9d74fb9c894eeae"
server_key_version: "encryptionkey@0"
I20260925 16:33:09.290776 3668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:33:09.310532 3668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:33:09.314033 3668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:33:09.314241 3668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:33:09.314926 3668 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:33:09.315765 3668 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:33:09.315837 3668 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:09.315909 3668 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:33:09.315961 3668 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:09.325440 3668 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43091
I20260925 16:33:09.325456 3796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43091 every 8 connection(s)
I20260925 16:33:09.326356 3668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
I20260925 16:33:09.334273 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 3668
I20260925 16:33:09.334437 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-1/wal/instance
I20260925 16:33:09.334807 392 external_mini_cluster.cc:1428] Setting key 9d7a7255b413098ea76e2420bdc80ce6
Sep 25 16:33:09 dist-test-slave-65pf krb5kdc[3427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:33:09.344167 3511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:09.328744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35209 (local address 127.0.98.62:35061)
0925 16:33:09.328927 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:09.328933 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:09.329538 (+ 605us) server_negotiation.cc:407] Connection header received
0925 16:33:09.331721 (+ 2183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:09.331727 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:09.331793 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:09.331906 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:09.333634 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:09.334469 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:09.336322 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:09.336633 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:09.336760 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:09.339799 (+ 3039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:09.339805 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:09.339811 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:09.339846 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:09.341672 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:09.342210 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:09.342216 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:09.342221 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:09.342323 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:09.342836 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:09.342841 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:09.342844 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:09.343115 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:09.343230 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:09.343829 (+ 599us) server_negotiation.cc:299] Negotiation successful
0925 16:33:09.343981 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260925 16:33:09.344996 3799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:09.328791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35061 (local address 127.0.98.2:35209)
0925 16:33:09.329357 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:09.329422 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:09.331420 (+ 1998us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:09.332141 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:09.332154 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:09.332728 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:09.333461 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:09.333482 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:09.334680 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:09.334686 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:09.335990 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:09.336003 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:09.336814 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:09.336821 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:09.337100 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:09.337813 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:09.337834 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:09.339618 (+ 1784us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:33:09.341808 (+ 2190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:09.341813 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:09.341826 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:09.342088 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:09.342441 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:09.342444 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:09.342446 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:09.342716 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:09.343263 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:33:09.343269 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:33:09.343547 (+ 278us) client_negotiation.cc:769] Sending connection context
0925 16:33:09.343719 (+ 172us) client_negotiation.cc:240] Negotiation successful
0925 16:33:09.344013 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":168,"threads_started":1}
I20260925 16:33:09.346374 3797 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35061
I20260925 16:33:09.346694 3797 heartbeater.cc:464] Registering TS with master...
I20260925 16:33:09.347260 3797 heartbeater.cc:511] Master 127.0.98.62:35061 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260925 16:33:09.348453 3472 ts_manager.cc:194] Registered new tserver with Master: 2988884eefe14f41a27e1496b1bf6a0a (127.0.98.2:43091)
I20260925 16:33:09.349040 3472 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35209
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 16:33:09.363256 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35061
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:33:09.478896 3802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:33:09.479173 3802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:33:09.479224 3802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:33:09.483291 3802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:33:09.483361 3802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:33:09.483421 3802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 16:33:09.489291 3802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:33:09.490397 3802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:33:09.491479 3802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:33:09.501457 3807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:09.501469 3810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:09.501570 3808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:09.502008 3802 server_base.cc:1034] running on GCE node
I20260925 16:33:09.502451 3802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:33:09.503146 3802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:33:09.504323 3802 hybrid_clock.cc:648] HybridClock initialized: now 1790353989504309 us; error 29 us; skew 500 ppm
Sep 25 16:33:09 dist-test-slave-65pf krb5kdc[3427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:33:09.507301 3802 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 16:33:09.508176 3802 webserver.cc:460] Webserver started at http://127.0.98.3:33905/ using document root <none> and password file <none>
I20260925 16:33:09.508706 3802 fs_manager.cc:359] Metadata directory not provided
I20260925 16:33:09.508769 3802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:33:09.508935 3802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:33:09.511042 3802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-2/data/instance:
uuid: "d17171f1cd88492ebaf31010ddc09e1e"
format_stamp: "Formatted at 2026-09-25 16:33:09 on dist-test-slave-65pf"
server_key: "b058d9d4266b84fbcf72cd797a1f8784"
server_key_iv: "e02f82ac0a8951295d21c6059fed228c"
server_key_version: "encryptionkey@0"
I20260925 16:33:09.511574 3802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d17171f1cd88492ebaf31010ddc09e1e"
format_stamp: "Formatted at 2026-09-25 16:33:09 on dist-test-slave-65pf"
server_key: "b058d9d4266b84fbcf72cd797a1f8784"
server_key_iv: "e02f82ac0a8951295d21c6059fed228c"
server_key_version: "encryptionkey@0"
I20260925 16:33:09.515425 3802 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 16:33:09.518543 3817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:09.519925 3802 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 16:33:09.520063 3802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-2/wal
uuid: "d17171f1cd88492ebaf31010ddc09e1e"
format_stamp: "Formatted at 2026-09-25 16:33:09 on dist-test-slave-65pf"
server_key: "b058d9d4266b84fbcf72cd797a1f8784"
server_key_iv: "e02f82ac0a8951295d21c6059fed228c"
server_key_version: "encryptionkey@0"
I20260925 16:33:09.520191 3802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:33:09.538156 3802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:33:09.541121 3802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:33:09.541330 3802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:33:09.541915 3802 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:33:09.542798 3802 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:33:09.542874 3802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:09.542948 3802 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:33:09.542990 3802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:09.552059 3802 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45783
I20260925 16:33:09.552080 3930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45783 every 8 connection(s)
I20260925 16:33:09.552973 3802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
I20260925 16:33:09.559979 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 3802
I20260925 16:33:09.560133 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353966814071-392-0/minicluster-data/ts-2/wal/instance
I20260925 16:33:09.560443 392 external_mini_cluster.cc:1428] Setting key 9a72f3fe0c41aed1e558e7535035adae
Sep 25 16:33:09 dist-test-slave-65pf krb5kdc[3427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:33:09.567241 3511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:09.554819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45643 (local address 127.0.98.62:35061)
0925 16:33:09.554995 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:09.554999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:09.555981 (+ 982us) server_negotiation.cc:407] Connection header received
0925 16:33:09.557299 (+ 1318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:09.557302 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:09.557350 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:09.557426 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:09.558560 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:09.559134 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:09.560257 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:09.560590 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:09.560700 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:09.563687 (+ 2987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:09.563691 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:09.563694 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:09.563721 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:09.565150 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:09.565648 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:09.565653 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:09.565655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:09.565742 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:09.566089 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:09.566094 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:09.566096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:09.566284 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:09.566394 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:09.566956 (+ 562us) server_negotiation.cc:299] Negotiation successful
0925 16:33:09.567080 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 16:33:09.568055 3933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:09.555335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35061 (local address 127.0.98.3:45643)
0925 16:33:09.555818 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:09.555862 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:09.557106 (+ 1244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:09.557588 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:09.557595 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:09.557944 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:09.558432 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:09.558446 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:09.559239 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:09.559241 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:09.560145 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:09.560156 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:09.560729 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:09.560731 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:09.560946 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:09.561605 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:09.561628 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:09.563481 (+ 1853us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:33:09.565260 (+ 1779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:09.565266 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:09.565281 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:09.565543 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:09.565841 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:09.565844 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:09.565846 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:09.565993 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:09.566390 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:33:09.566397 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:33:09.566715 (+ 318us) client_negotiation.cc:769] Sending connection context
0925 16:33:09.566903 (+ 188us) client_negotiation.cc:240] Negotiation successful
0925 16:33:09.567235 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":117,"threads_started":1}
I20260925 16:33:09.569132 3931 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35061
I20260925 16:33:09.569343 3931 heartbeater.cc:464] Registering TS with master...
I20260925 16:33:09.569780 3931 heartbeater.cc:511] Master 127.0.98.62:35061 requested a full tablet report, sending...
I20260925 16:33:09.570770 3472 ts_manager.cc:194] Registered new tserver with Master: d17171f1cd88492ebaf31010ddc09e1e (127.0.98.3:45783)
I20260925 16:33:09.571341 3472 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45643
I20260925 16:33:09.574260 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 16:33:09 dist-test-slave-65pf krb5kdc[3427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353989, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 25 16:33:09 dist-test-slave-65pf krb5kdc[3427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353989, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:33:09.599561 3511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:09.591172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35310 (local address 127.0.98.62:35061)
0925 16:33:09.591316 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:09.591319 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:09.591479 (+ 160us) server_negotiation.cc:407] Connection header received
0925 16:33:09.591596 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:09.591598 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:09.591643 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:09.591722 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:09.592428 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:09.592943 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:09.593772 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:09.593997 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:09.594071 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:09.596586 (+ 2515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:09.596589 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:09.596591 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:09.596616 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:09.598091 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:09.598444 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:09.598447 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:09.598448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:09.598518 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:09.598825 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:09.598827 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:09.598829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:09.599047 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:09.599150 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:09.599343 (+ 193us) server_negotiation.cc:299] Negotiation successful
0925 16:33:09.599438 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260925 16:33:09.599969 3472 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:35310
I20260925 16:33:09.600966 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 3534
I20260925 16:33:09.607141 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 3668
I20260925 16:33:09.612437 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 3802
I20260925 16:33:09.617847 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 3440
2026-09-25T16:33:09Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2110 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:33:09 dist-test-slave-65pf krb5kdc[3943](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:33:09 dist-test-slave-65pf krb5kdc[3943](info): set up 2 sockets
Sep 25 16:33:09 dist-test-slave-65pf krb5kdc[3943](info): commencing operation
krb5kdc: starting...
W20260925 16:33:10.674729 392 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/krb5kdc/test-user.keytab.
Sep 25 16:33:10 dist-test-slave-65pf krb5kdc[3943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:33:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:33:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 16:33:10.797492 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46661
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46661
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:33:10.908275 3956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:33:10.908617 3956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:33:10.908690 3956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:33:10.912849 3956 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:33:10.912947 3956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:33:10.912986 3956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:33:10.913048 3956 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:33:10.913079 3956 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:33:10.918495 3956 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46661
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:33:10.919631 3956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:33:10.920612 3956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:33:10.929112 3961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:10.929112 3962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:10.929338 3956 server_base.cc:1034] running on GCE node
W20260925 16:33:10.929144 3964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:10.930042 3956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:33:10.931252 3956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:33:10.932441 3956 hybrid_clock.cc:648] HybridClock initialized: now 1790353990932428 us; error 27 us; skew 500 ppm
Sep 25 16:33:10 dist-test-slave-65pf krb5kdc[3943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:33:10.935313 3956 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 16:33:10.936282 3956 webserver.cc:460] Webserver started at http://127.0.98.62:37211/ using document root <none> and password file <none>
I20260925 16:33:10.936866 3956 fs_manager.cc:359] Metadata directory not provided
I20260925 16:33:10.936960 3956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:33:10.937202 3956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:33:10.939378 3956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/minicluster-data/master-0/data/instance:
uuid: "13f136696f2c4a9995b53d6323744cc3"
format_stamp: "Formatted at 2026-09-25 16:33:10 on dist-test-slave-65pf"
server_key: "d3d462f6f0647c6062f081da0463a610"
server_key_iv: "a3b22e7f1ee1e6bf7fb3eb4af925ff8e"
server_key_version: "encryptionkey@0"
I20260925 16:33:10.939958 3956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/minicluster-data/master-0/wal/instance:
uuid: "13f136696f2c4a9995b53d6323744cc3"
format_stamp: "Formatted at 2026-09-25 16:33:10 on dist-test-slave-65pf"
server_key: "d3d462f6f0647c6062f081da0463a610"
server_key_iv: "a3b22e7f1ee1e6bf7fb3eb4af925ff8e"
server_key_version: "encryptionkey@0"
I20260925 16:33:10.943687 3956 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 16:33:10.946084 3971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:10.947431 3956 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:33:10.947578 3956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/minicluster-data/master-0/wal
uuid: "13f136696f2c4a9995b53d6323744cc3"
format_stamp: "Formatted at 2026-09-25 16:33:10 on dist-test-slave-65pf"
server_key: "d3d462f6f0647c6062f081da0463a610"
server_key_iv: "a3b22e7f1ee1e6bf7fb3eb4af925ff8e"
server_key_version: "encryptionkey@0"
I20260925 16:33:10.947728 3956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:33:10.971516 3956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:33:10.975147 3956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:33:10.975378 3956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:33:10.982484 3956 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46661
I20260925 16:33:10.982510 4023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46661 every 8 connection(s)
I20260925 16:33:10.983583 3956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:33:10.984424 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 3956
I20260925 16:33:10.984541 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/minicluster-data/master-0/wal/instance
I20260925 16:33:10.984843 392 external_mini_cluster.cc:1428] Setting key f9fe48dcda4e564a48daabf02e498c3a
I20260925 16:33:10.987035 4024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:33:10.992681 4024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13f136696f2c4a9995b53d6323744cc3: Bootstrap starting.
Sep 25 16:33:10 dist-test-slave-65pf krb5kdc[3943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:33:10.995188 4024 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13f136696f2c4a9995b53d6323744cc3: Neither blocks nor log segments found. Creating new log.
I20260925 16:33:10.995954 4024 log.cc:824] T 00000000000000000000000000000000 P 13f136696f2c4a9995b53d6323744cc3: Log is configured to *not* fsync() on all Append() calls
I20260925 16:33:10.998374 4024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13f136696f2c4a9995b53d6323744cc3: No bootstrap required, opened a new log
I20260925 16:33:10.999341 4027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:10.986255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41114 (local address 127.0.98.62:46661)
0925 16:33:10.986817 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:10.986834 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:10.986872 (+ 38us) server_negotiation.cc:407] Connection header received
0925 16:33:10.987579 (+ 707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:10.987597 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:10.987934 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:10.988287 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:10.989107 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:10.990089 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:10.991062 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:10.991549 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:10.991719 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:10.994261 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:10.994266 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:10.994280 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:10.994308 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:10.996106 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:10.996620 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:10.996625 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:10.996629 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:10.996743 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:10.997049 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:10.997052 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:10.997054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:10.997566 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:10.997764 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:10.998004 (+ 240us) server_negotiation.cc:299] Negotiation successful
0925 16:33:10.998314 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":148,"threads_started":1}
I20260925 16:33:11.002478 4024 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13f136696f2c4a9995b53d6323744cc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13f136696f2c4a9995b53d6323744cc3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46661 } }
I20260925 16:33:11.002780 4024 raft_consensus.cc:374] T 00000000000000000000000000000000 P 13f136696f2c4a9995b53d6323744cc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:33:11.002823 4024 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13f136696f2c4a9995b53d6323744cc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13f136696f2c4a9995b53d6323744cc3, State: Initialized, Role: FOLLOWER
I20260925 16:33:11.003193 4024 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13f136696f2c4a9995b53d6323744cc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13f136696f2c4a9995b53d6323744cc3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46661 } }
I20260925 16:33:11.003299 4024 raft_consensus.cc:388] T 00000000000000000000000000000000 P 13f136696f2c4a9995b53d6323744cc3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:33:11.003341 4024 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13f136696f2c4a9995b53d6323744cc3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:33:11.003414 4024 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13f136696f2c4a9995b53d6323744cc3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:33:11.004302 4024 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13f136696f2c4a9995b53d6323744cc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13f136696f2c4a9995b53d6323744cc3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46661 } }
I20260925 16:33:11.004608 4024 leader_election.cc:302] T 00000000000000000000000000000000 P 13f136696f2c4a9995b53d6323744cc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13f136696f2c4a9995b53d6323744cc3; no voters:
I20260925 16:33:11.004866 4024 leader_election.cc:288] T 00000000000000000000000000000000 P 13f136696f2c4a9995b53d6323744cc3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:33:11.004999 4029 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13f136696f2c4a9995b53d6323744cc3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:33:11.005285 4029 raft_consensus.cc:686] T 00000000000000000000000000000000 P 13f136696f2c4a9995b53d6323744cc3 [term 1 LEADER]: Becoming Leader. State: Replica: 13f136696f2c4a9995b53d6323744cc3, State: Running, Role: LEADER
I20260925 16:33:11.005610 4029 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13f136696f2c4a9995b53d6323744cc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13f136696f2c4a9995b53d6323744cc3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46661 } }
I20260925 16:33:11.006139 4024 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13f136696f2c4a9995b53d6323744cc3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:33:11.007306 4030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13f136696f2c4a9995b53d6323744cc3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "13f136696f2c4a9995b53d6323744cc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13f136696f2c4a9995b53d6323744cc3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46661 } } }
I20260925 16:33:11.007468 4030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13f136696f2c4a9995b53d6323744cc3 [sys.catalog]: This master's current role is: LEADER
I20260925 16:33:11.007813 4031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13f136696f2c4a9995b53d6323744cc3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13f136696f2c4a9995b53d6323744cc3. Latest consensus state: current_term: 1 leader_uuid: "13f136696f2c4a9995b53d6323744cc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13f136696f2c4a9995b53d6323744cc3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46661 } } }
I20260925 16:33:11.007974 4031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13f136696f2c4a9995b53d6323744cc3 [sys.catalog]: This master's current role is: LEADER
I20260925 16:33:11.008364 4038 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:33:11.011333 4038 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:33:11.016932 4038 catalog_manager.cc:1347] Generated new cluster ID: c5f93fe58c394e0aabfcbd28233139ce
I20260925 16:33:11.017009 4038 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:33:11.023152 4038 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:33:11.024331 4038 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:33:11.033143 4038 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 13f136696f2c4a9995b53d6323744cc3: Generated new TSK 0
I20260925 16:33:11.034039 4038 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 16:33:11.065256 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46661
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:33:11.177222 4050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:33:11.177491 4050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:33:11.177534 4050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:33:11.181700 4050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:33:11.181757 4050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:33:11.181815 4050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 16:33:11.187105 4050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:33:11.188191 4050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:33:11.189246 4050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 16:33:11.198784 4050 server_base.cc:1034] running on GCE node
W20260925 16:33:11.198731 4058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:11.198731 4055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:11.198844 4056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:11.199453 4050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:33:11.200078 4050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:33:11.201238 4050 hybrid_clock.cc:648] HybridClock initialized: now 1790353991201223 us; error 28 us; skew 500 ppm
Sep 25 16:33:11 dist-test-slave-65pf krb5kdc[3943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353991, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:33:11.204061 4050 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 16:33:11.204969 4050 webserver.cc:460] Webserver started at http://127.0.98.1:43479/ using document root <none> and password file <none>
I20260925 16:33:11.205534 4050 fs_manager.cc:359] Metadata directory not provided
I20260925 16:33:11.205595 4050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:33:11.205838 4050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:33:11.208118 4050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/minicluster-data/ts-0/data/instance:
uuid: "1aa10f986b79414e8163461e0035fcd1"
format_stamp: "Formatted at 2026-09-25 16:33:11 on dist-test-slave-65pf"
server_key: "13d57b2ea8bb54338b1b6a7e4f0442b5"
server_key_iv: "8380c6d010e52423f171283732ef1997"
server_key_version: "encryptionkey@0"
I20260925 16:33:11.208726 4050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1aa10f986b79414e8163461e0035fcd1"
format_stamp: "Formatted at 2026-09-25 16:33:11 on dist-test-slave-65pf"
server_key: "13d57b2ea8bb54338b1b6a7e4f0442b5"
server_key_iv: "8380c6d010e52423f171283732ef1997"
server_key_version: "encryptionkey@0"
I20260925 16:33:11.212494 4050 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260925 16:33:11.214946 4065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:11.216188 4050 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 16:33:11.216308 4050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/minicluster-data/ts-0/wal
uuid: "1aa10f986b79414e8163461e0035fcd1"
format_stamp: "Formatted at 2026-09-25 16:33:11 on dist-test-slave-65pf"
server_key: "13d57b2ea8bb54338b1b6a7e4f0442b5"
server_key_iv: "8380c6d010e52423f171283732ef1997"
server_key_version: "encryptionkey@0"
I20260925 16:33:11.216408 4050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:33:11.237738 4050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:33:11.240880 4050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:33:11.241091 4050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:33:11.241720 4050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:33:11.242548 4050 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:33:11.242646 4050 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:11.242736 4050 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:33:11.242769 4050 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:11.251726 4050 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37535
I20260925 16:33:11.251813 4178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37535 every 8 connection(s)
I20260925 16:33:11.252661 4050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
I20260925 16:33:11.261749 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 4050
I20260925 16:33:11.261907 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/minicluster-data/ts-0/wal/instance
I20260925 16:33:11.262203 392 external_mini_cluster.cc:1428] Setting key 39ff510482917e19a1314054652e689f
Sep 25 16:33:11 dist-test-slave-65pf krb5kdc[3943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353991, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:33:11.266685 4027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:11.254441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52825 (local address 127.0.98.62:46661)
0925 16:33:11.254619 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:11.254625 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:11.255562 (+ 937us) server_negotiation.cc:407] Connection header received
0925 16:33:11.256956 (+ 1394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:11.256959 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:11.257013 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:11.257125 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:11.258313 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:11.258921 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:11.260060 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:11.260300 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:11.260413 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:11.263018 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:11.263021 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:11.263024 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:11.263048 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:11.264616 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:11.265138 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:11.265143 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:11.265145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:11.265227 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:11.265564 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:11.265569 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:11.265571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:11.265749 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:11.265864 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:11.266364 (+ 500us) server_negotiation.cc:299] Negotiation successful
0925 16:33:11.266490 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 16:33:11.267511 4181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:11.254966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46661 (local address 127.0.98.1:52825)
0925 16:33:11.255406 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:11.255449 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:11.256740 (+ 1291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:11.257301 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:11.257308 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:11.257674 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:11.258154 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:11.258167 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:11.259048 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:11.259058 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:11.259933 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:11.259942 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:11.260456 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:11.260460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:11.260623 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:11.261179 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:11.261195 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:11.262892 (+ 1697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:33:11.264740 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:11.264745 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:11.264760 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:11.265033 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:11.265326 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:11.265329 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:11.265331 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:11.265468 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:11.265859 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:33:11.265866 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:33:11.266116 (+ 250us) client_negotiation.cc:769] Sending connection context
0925 16:33:11.266324 (+ 208us) client_negotiation.cc:240] Negotiation successful
0925 16:33:11.266679 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":100,"threads_started":1}
I20260925 16:33:11.268786 4179 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46661
I20260925 16:33:11.269042 4179 heartbeater.cc:464] Registering TS with master...
I20260925 16:33:11.269531 4179 heartbeater.cc:511] Master 127.0.98.62:46661 requested a full tablet report, sending...
I20260925 16:33:11.271071 3988 ts_manager.cc:194] Registered new tserver with Master: 1aa10f986b79414e8163461e0035fcd1 (127.0.98.1:37535)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 16:33:11.272220 3988 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52825
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 16:33:11.287811 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46661
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:33:11.397902 4184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:33:11.398197 4184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:33:11.398278 4184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:33:11.402334 4184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:33:11.402436 4184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:33:11.402526 4184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 16:33:11.408082 4184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:33:11.409173 4184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:33:11.410200 4184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:33:11.419518 4189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:11.419687 4184 server_base.cc:1034] running on GCE node
W20260925 16:33:11.419514 4192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:11.419515 4190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:11.420223 4184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:33:11.420912 4184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:33:11.422070 4184 hybrid_clock.cc:648] HybridClock initialized: now 1790353991422048 us; error 37 us; skew 500 ppm
Sep 25 16:33:11 dist-test-slave-65pf krb5kdc[3943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353991, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:33:11.424734 4184 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 16:33:11.425598 4184 webserver.cc:460] Webserver started at http://127.0.98.2:43223/ using document root <none> and password file <none>
I20260925 16:33:11.426189 4184 fs_manager.cc:359] Metadata directory not provided
I20260925 16:33:11.426288 4184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:33:11.426519 4184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:33:11.428550 4184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/minicluster-data/ts-1/data/instance:
uuid: "0ca2213d4e3a432abe43b76cc23d408c"
format_stamp: "Formatted at 2026-09-25 16:33:11 on dist-test-slave-65pf"
server_key: "25e52e4425d752bdfcf126ae2edfe6a3"
server_key_iv: "2d4ef42ae2ef01594e35ca52b7b11398"
server_key_version: "encryptionkey@0"
I20260925 16:33:11.429155 4184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0ca2213d4e3a432abe43b76cc23d408c"
format_stamp: "Formatted at 2026-09-25 16:33:11 on dist-test-slave-65pf"
server_key: "25e52e4425d752bdfcf126ae2edfe6a3"
server_key_iv: "2d4ef42ae2ef01594e35ca52b7b11398"
server_key_version: "encryptionkey@0"
I20260925 16:33:11.432861 4184 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 16:33:11.435297 4199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:11.436520 4184 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:33:11.436656 4184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/minicluster-data/ts-1/wal
uuid: "0ca2213d4e3a432abe43b76cc23d408c"
format_stamp: "Formatted at 2026-09-25 16:33:11 on dist-test-slave-65pf"
server_key: "25e52e4425d752bdfcf126ae2edfe6a3"
server_key_iv: "2d4ef42ae2ef01594e35ca52b7b11398"
server_key_version: "encryptionkey@0"
I20260925 16:33:11.436784 4184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:33:11.447597 4184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:33:11.450655 4184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:33:11.450871 4184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:33:11.451550 4184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:33:11.452384 4184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:33:11.452455 4184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:11.452519 4184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:33:11.452567 4184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:11.461179 4184 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45927
I20260925 16:33:11.461207 4312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45927 every 8 connection(s)
I20260925 16:33:11.462064 4184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
I20260925 16:33:11.463654 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 4184
I20260925 16:33:11.463786 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/minicluster-data/ts-1/wal/instance
I20260925 16:33:11.464094 392 external_mini_cluster.cc:1428] Setting key 0fcf046e0ffd7897d6db0c8404f5cc89
Sep 25 16:33:11 dist-test-slave-65pf krb5kdc[3943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353991, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260925 16:33:11.476367 4027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:11.463862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32909 (local address 127.0.98.62:46661)
0925 16:33:11.464021 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:11.464027 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:11.464952 (+ 925us) server_negotiation.cc:407] Connection header received
0925 16:33:11.466371 (+ 1419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:11.466375 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:11.466429 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:11.466535 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:11.467756 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:11.468287 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:11.469527 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:11.469758 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:11.469870 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:11.472600 (+ 2730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:11.472604 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:11.472607 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:11.472636 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:11.474173 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:11.474715 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:11.474718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:11.474719 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:11.474794 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:11.475194 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:11.475196 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:11.475197 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:11.475378 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:11.475489 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:11.476062 (+ 573us) server_negotiation.cc:299] Negotiation successful
0925 16:33:11.476197 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 16:33:11.477240 4315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:11.464344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46661 (local address 127.0.98.2:32909)
0925 16:33:11.464799 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:11.464851 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:11.466143 (+ 1292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:11.466752 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:11.466760 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:11.467112 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:11.467609 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:11.467622 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:11.468413 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:11.468417 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:11.469403 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:11.469412 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:11.469910 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:11.469914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:11.470080 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:11.470859 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:11.470876 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:11.472392 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:33:11.474283 (+ 1891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:11.474289 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:11.474302 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:11.474568 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:11.474915 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:11.474918 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:11.474920 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:11.475104 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:11.475482 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:33:11.475487 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:33:11.475803 (+ 316us) client_negotiation.cc:769] Sending connection context
0925 16:33:11.476006 (+ 203us) client_negotiation.cc:240] Negotiation successful
0925 16:33:11.476370 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":122,"threads_started":1}
I20260925 16:33:11.478530 4313 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46661
I20260925 16:33:11.478818 4313 heartbeater.cc:464] Registering TS with master...
I20260925 16:33:11.479454 4313 heartbeater.cc:511] Master 127.0.98.62:46661 requested a full tablet report, sending...
I20260925 16:33:11.480763 3988 ts_manager.cc:194] Registered new tserver with Master: 0ca2213d4e3a432abe43b76cc23d408c (127.0.98.2:45927)
I20260925 16:33:11.481681 3988 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:32909
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 16:33:11.491786 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46661
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:33:11.604389 4318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:33:11.604672 4318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:33:11.604717 4318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:33:11.609195 4318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:33:11.609265 4318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:33:11.609365 4318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 16:33:11.614799 4318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:33:11.616098 4318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:33:11.617166 4318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:33:11.627200 4323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:11.627197 4324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:11.627233 4318 server_base.cc:1034] running on GCE node
W20260925 16:33:11.627450 4326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:11.627840 4318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:33:11.628500 4318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:33:11.629660 4318 hybrid_clock.cc:648] HybridClock initialized: now 1790353991629633 us; error 41 us; skew 500 ppm
Sep 25 16:33:11 dist-test-slave-65pf krb5kdc[3943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353991, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:33:11.632489 4318 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 16:33:11.633301 4318 webserver.cc:460] Webserver started at http://127.0.98.3:46135/ using document root <none> and password file <none>
I20260925 16:33:11.633819 4318 fs_manager.cc:359] Metadata directory not provided
I20260925 16:33:11.633876 4318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:33:11.634035 4318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:33:11.636082 4318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/minicluster-data/ts-2/data/instance:
uuid: "fa6d896f905c4470b10990cea538966a"
format_stamp: "Formatted at 2026-09-25 16:33:11 on dist-test-slave-65pf"
server_key: "475c18a4834be874fdbc24dfcc3da6dc"
server_key_iv: "13de3d9f068367f97fce866d63cd7b7c"
server_key_version: "encryptionkey@0"
I20260925 16:33:11.636600 4318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/minicluster-data/ts-2/wal/instance:
uuid: "fa6d896f905c4470b10990cea538966a"
format_stamp: "Formatted at 2026-09-25 16:33:11 on dist-test-slave-65pf"
server_key: "475c18a4834be874fdbc24dfcc3da6dc"
server_key_iv: "13de3d9f068367f97fce866d63cd7b7c"
server_key_version: "encryptionkey@0"
I20260925 16:33:11.640508 4318 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 16:33:11.642976 4333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:11.644212 4318 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:33:11.644349 4318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/minicluster-data/ts-2/wal
uuid: "fa6d896f905c4470b10990cea538966a"
format_stamp: "Formatted at 2026-09-25 16:33:11 on dist-test-slave-65pf"
server_key: "475c18a4834be874fdbc24dfcc3da6dc"
server_key_iv: "13de3d9f068367f97fce866d63cd7b7c"
server_key_version: "encryptionkey@0"
I20260925 16:33:11.644465 4318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:33:11.656855 4318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:33:11.660277 4318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:33:11.660490 4318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:33:11.661084 4318 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:33:11.661903 4318 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:33:11.661973 4318 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:11.662042 4318 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:33:11.662092 4318 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:11.670769 4318 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34231
I20260925 16:33:11.670794 4446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34231 every 8 connection(s)
I20260925 16:33:11.671672 4318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
I20260925 16:33:11.678225 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 4318
I20260925 16:33:11.678347 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353966814071-392-0/minicluster-data/ts-2/wal/instance
I20260925 16:33:11.678651 392 external_mini_cluster.cc:1428] Setting key 6d76328ea961c25ed7960ef5e6178cf6
Sep 25 16:33:11 dist-test-slave-65pf krb5kdc[3943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353991, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:33:11.685104 4027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:11.673464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56963 (local address 127.0.98.62:46661)
0925 16:33:11.673628 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:11.673633 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:11.674468 (+ 835us) server_negotiation.cc:407] Connection header received
0925 16:33:11.675836 (+ 1368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:11.675839 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:11.675885 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:11.675960 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:11.677066 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:11.677584 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:11.679108 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:11.679415 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:11.679511 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:11.681944 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:11.681947 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:11.681949 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:11.681973 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:11.683471 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:11.683896 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:11.683898 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:11.683900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:11.683968 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:11.684244 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:11.684246 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:11.684247 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:11.684406 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:11.684476 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:11.684896 (+ 420us) server_negotiation.cc:299] Negotiation successful
0925 16:33:11.684988 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 16:33:11.685896 4449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:11.673896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46661 (local address 127.0.98.3:56963)
0925 16:33:11.674316 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:11.674359 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:11.675644 (+ 1285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:11.676117 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:11.676123 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:11.676462 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:11.676936 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:11.676950 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:11.677684 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:11.677686 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:11.678985 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:11.678994 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:11.679548 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:11.679552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:11.679729 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:11.680312 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:11.680332 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:11.681802 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:33:11.683562 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:11.683566 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:11.683578 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:11.683814 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:11.684044 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:11.684046 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:11.684047 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:11.684170 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:11.684488 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:33:11.684493 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:33:11.684704 (+ 211us) client_negotiation.cc:769] Sending connection context
0925 16:33:11.684852 (+ 148us) client_negotiation.cc:240] Negotiation successful
0925 16:33:11.685086 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":96,"threads_started":1}
I20260925 16:33:11.687006 4447 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46661
I20260925 16:33:11.687220 4447 heartbeater.cc:464] Registering TS with master...
I20260925 16:33:11.687654 4447 heartbeater.cc:511] Master 127.0.98.62:46661 requested a full tablet report, sending...
I20260925 16:33:11.688577 3988 ts_manager.cc:194] Registered new tserver with Master: fa6d896f905c4470b10990cea538966a (127.0.98.3:34231)
I20260925 16:33:11.689100 3988 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56963
I20260925 16:33:11.692332 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 16:33:11 dist-test-slave-65pf krb5kdc[3943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 16:33:11 dist-test-slave-65pf krb5kdc[3943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 16:33:11.717615 4454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:11.708275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60480 (local address 127.0.98.1:37535)
0925 16:33:11.708601 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:11.708611 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:11.708634 (+ 23us) server_negotiation.cc:407] Connection header received
0925 16:33:11.708692 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:11.708697 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:11.708891 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:11.709013 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:11.709667 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:11.710443 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:11.711290 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:11.711660 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:11.711752 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:11.714133 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:11.714142 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:11.714163 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:11.714189 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:11.715938 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:11.716336 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:11.716342 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:11.716344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:11.716423 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:11.716650 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:11.716653 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:11.716654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:11.716923 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:11.717094 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:11.717240 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 16:33:11.717401 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":116,"threads_started":1}
Sep 25 16:33:11 dist-test-slave-65pf krb5kdc[3943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:33:11.729390 4027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:11.720773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41130 (local address 127.0.98.62:46661)
0925 16:33:11.720917 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:11.720924 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:11.720944 (+ 20us) server_negotiation.cc:407] Connection header received
0925 16:33:11.721095 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:11.721098 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:11.721146 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:11.721236 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:11.721827 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:11.722362 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:11.723228 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:11.723471 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:11.723578 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:11.726164 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:11.726173 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:11.726175 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:11.726200 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:11.727651 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:11.728091 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:11.728094 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:11.728096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:11.728185 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:11.728468 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:11.728471 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:11.728473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:11.728706 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:11.728844 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:11.729087 (+ 243us) server_negotiation.cc:299] Negotiation successful
0925 16:33:11.729221 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260925 16:33:11.729946 3988 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41130
I20260925 16:33:11.730772 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 4050
I20260925 16:33:11.736935 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 4184
I20260925 16:33:11.742219 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 4318
I20260925 16:33:11.747608 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 3956
2026-09-25T16:33:11Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2129 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:33:11 dist-test-slave-65pf krb5kdc[4459](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:33:11 dist-test-slave-65pf krb5kdc[4459](info): set up 2 sockets
Sep 25 16:33:11 dist-test-slave-65pf krb5kdc[4459](info): commencing operation
krb5kdc: starting...
W20260925 16:33:12.790961 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/krb5kdc/test-user.keytab.
Sep 25 16:33:12 dist-test-slave-65pf krb5kdc[4459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:33:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:33:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 16:33:12.913141 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39347
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39347
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:33:13.023068 4472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:33:13.023371 4472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:33:13.023447 4472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:33:13.027477 4472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:33:13.027565 4472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:33:13.027617 4472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:33:13.027674 4472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:33:13.027712 4472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:33:13.033995 4472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39347
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:33:13.035164 4472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:33:13.036191 4472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:33:13.044306 4477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:13.044369 4478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:13.044335 4480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:13.044500 4472 server_base.cc:1034] running on GCE node
I20260925 16:33:13.045117 4472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:33:13.046221 4472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:33:13.047406 4472 hybrid_clock.cc:648] HybridClock initialized: now 1790353993047393 us; error 26 us; skew 500 ppm
Sep 25 16:33:13 dist-test-slave-65pf krb5kdc[4459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353993, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:33:13.056687 4472 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 16:33:13.057410 4472 fs_manager.cc:359] Metadata directory not provided
I20260925 16:33:13.057471 4472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:33:13.057632 4472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:33:13.059752 4472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/minicluster-data/master-0/data/instance:
uuid: "166ff127944b437baee01105a5b9094a"
format_stamp: "Formatted at 2026-09-25 16:33:13 on dist-test-slave-65pf"
server_key: "49f475bb460ff1846c14440472d238dd"
server_key_iv: "5511c8f46db53d11691153330ae07668"
server_key_version: "encryptionkey@0"
I20260925 16:33:13.060266 4472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/minicluster-data/master-0/wal/instance:
uuid: "166ff127944b437baee01105a5b9094a"
format_stamp: "Formatted at 2026-09-25 16:33:13 on dist-test-slave-65pf"
server_key: "49f475bb460ff1846c14440472d238dd"
server_key_iv: "5511c8f46db53d11691153330ae07668"
server_key_version: "encryptionkey@0"
I20260925 16:33:13.064350 4472 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 16:33:13.066849 4486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:13.068116 4472 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:33:13.068218 4472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/minicluster-data/master-0/wal
uuid: "166ff127944b437baee01105a5b9094a"
format_stamp: "Formatted at 2026-09-25 16:33:13 on dist-test-slave-65pf"
server_key: "49f475bb460ff1846c14440472d238dd"
server_key_iv: "5511c8f46db53d11691153330ae07668"
server_key_version: "encryptionkey@0"
I20260925 16:33:13.068306 4472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:33:13.102197 4472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:33:13.105211 4472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:33:13.105377 4472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:33:13.112650 4472 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39347
I20260925 16:33:13.112911 4538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39347 every 8 connection(s)
I20260925 16:33:13.113384 4472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:33:13.116384 4539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:33:13.119860 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 4472
I20260925 16:33:13.119966 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/minicluster-data/master-0/wal/instance
I20260925 16:33:13.120242 392 external_mini_cluster.cc:1428] Setting key 63de5f916c25dbae463e6e2e58f812f7
I20260925 16:33:13.122534 4539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 166ff127944b437baee01105a5b9094a: Bootstrap starting.
I20260925 16:33:13.125241 4539 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 166ff127944b437baee01105a5b9094a: Neither blocks nor log segments found. Creating new log.
I20260925 16:33:13.126072 4539 log.cc:824] T 00000000000000000000000000000000 P 166ff127944b437baee01105a5b9094a: Log is configured to *not* fsync() on all Append() calls
I20260925 16:33:13.128702 4539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 166ff127944b437baee01105a5b9094a: No bootstrap required, opened a new log
Sep 25 16:33:13 dist-test-slave-65pf krb5kdc[4459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:33:13.132159 4539 raft_consensus.cc:348] T 00000000000000000000000000000000 P 166ff127944b437baee01105a5b9094a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "166ff127944b437baee01105a5b9094a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39347 } }
I20260925 16:33:13.132408 4539 raft_consensus.cc:374] T 00000000000000000000000000000000 P 166ff127944b437baee01105a5b9094a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:33:13.132505 4539 raft_consensus.cc:729] T 00000000000000000000000000000000 P 166ff127944b437baee01105a5b9094a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 166ff127944b437baee01105a5b9094a, State: Initialized, Role: FOLLOWER
I20260925 16:33:13.132993 4539 consensus_queue.cc:260] T 00000000000000000000000000000000 P 166ff127944b437baee01105a5b9094a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "166ff127944b437baee01105a5b9094a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39347 } }
I20260925 16:33:13.133159 4539 raft_consensus.cc:388] T 00000000000000000000000000000000 P 166ff127944b437baee01105a5b9094a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:33:13.133266 4539 raft_consensus.cc:482] T 00000000000000000000000000000000 P 166ff127944b437baee01105a5b9094a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:33:13.133386 4539 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 166ff127944b437baee01105a5b9094a [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:33:13.134301 4539 raft_consensus.cc:504] T 00000000000000000000000000000000 P 166ff127944b437baee01105a5b9094a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "166ff127944b437baee01105a5b9094a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39347 } }
I20260925 16:33:13.134723 4539 leader_election.cc:302] T 00000000000000000000000000000000 P 166ff127944b437baee01105a5b9094a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 166ff127944b437baee01105a5b9094a; no voters:
I20260925 16:33:13.135059 4539 leader_election.cc:288] T 00000000000000000000000000000000 P 166ff127944b437baee01105a5b9094a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:33:13.135159 4544 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 166ff127944b437baee01105a5b9094a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:33:13.135356 4544 raft_consensus.cc:686] T 00000000000000000000000000000000 P 166ff127944b437baee01105a5b9094a [term 1 LEADER]: Becoming Leader. State: Replica: 166ff127944b437baee01105a5b9094a, State: Running, Role: LEADER
I20260925 16:33:13.135699 4542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:13.121689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59446 (local address 127.0.98.62:39347)
0925 16:33:13.122119 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:13.122139 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:13.122181 (+ 42us) server_negotiation.cc:407] Connection header received
0925 16:33:13.123010 (+ 829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:13.123033 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:13.123440 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:13.123819 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:13.124441 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:13.125950 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:13.126888 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:13.127373 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:13.127499 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:13.129904 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:13.129913 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:13.129933 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:13.129964 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:13.132253 (+ 2289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:13.132755 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:13.132761 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:13.132766 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:13.132876 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:13.133252 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:13.133256 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:13.133257 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:13.133768 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:13.133985 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:13.134312 (+ 327us) server_negotiation.cc:299] Negotiation successful
0925 16:33:13.134668 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":111,"threads_started":1}
I20260925 16:33:13.135748 4544 consensus_queue.cc:237] T 00000000000000000000000000000000 P 166ff127944b437baee01105a5b9094a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "166ff127944b437baee01105a5b9094a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39347 } }
I20260925 16:33:13.136374 4539 sys_catalog.cc:564] T 00000000000000000000000000000000 P 166ff127944b437baee01105a5b9094a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:33:13.136916 4544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 166ff127944b437baee01105a5b9094a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 166ff127944b437baee01105a5b9094a. Latest consensus state: current_term: 1 leader_uuid: "166ff127944b437baee01105a5b9094a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "166ff127944b437baee01105a5b9094a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39347 } } }
I20260925 16:33:13.137116 4544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 166ff127944b437baee01105a5b9094a [sys.catalog]: This master's current role is: LEADER
I20260925 16:33:13.137418 4545 sys_catalog.cc:455] T 00000000000000000000000000000000 P 166ff127944b437baee01105a5b9094a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "166ff127944b437baee01105a5b9094a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "166ff127944b437baee01105a5b9094a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39347 } } }
I20260925 16:33:13.137511 4545 sys_catalog.cc:458] T 00000000000000000000000000000000 P 166ff127944b437baee01105a5b9094a [sys.catalog]: This master's current role is: LEADER
I20260925 16:33:13.137957 4552 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:33:13.141311 4552 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:33:13.147415 4552 catalog_manager.cc:1347] Generated new cluster ID: 68270796fed04f058ffff73419721f66
I20260925 16:33:13.147498 4552 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:33:13.162464 4552 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:33:13.163450 4552 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:33:13.168104 4552 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 166ff127944b437baee01105a5b9094a: Generated new TSK 0
I20260925 16:33:13.168797 4552 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 16:33:13.199591 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39347
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:33:13.332805 4565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:33:13.333096 4565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:33:13.333180 4565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:33:13.337441 4565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:33:13.337508 4565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:33:13.337620 4565 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 16:33:13.343127 4565 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4565
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:33:13.344282 4565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:33:13.345366 4565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:33:13.354941 4570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:13.354980 4573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:13.355118 4565 server_base.cc:1034] running on GCE node
W20260925 16:33:13.354961 4571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:13.355692 4565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:33:13.356349 4565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:33:13.357517 4565 hybrid_clock.cc:648] HybridClock initialized: now 1790353993357488 us; error 40 us; skew 500 ppm
Sep 25 16:33:13 dist-test-slave-65pf krb5kdc[4459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353993, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:33:13.360361 4565 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 16:33:13.361204 4565 fs_manager.cc:359] Metadata directory not provided
I20260925 16:33:13.361269 4565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:33:13.361486 4565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:33:13.363663 4565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/minicluster-data/ts-0/data/instance:
uuid: "00d60dce358c47209ac0685fd4243be9"
format_stamp: "Formatted at 2026-09-25 16:33:13 on dist-test-slave-65pf"
server_key: "2dc5f509fc48f48659cf7731107be43d"
server_key_iv: "54d004bd8c4bc838e024efaef91fb7c1"
server_key_version: "encryptionkey@0"
I20260925 16:33:13.364220 4565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/minicluster-data/ts-0/wal/instance:
uuid: "00d60dce358c47209ac0685fd4243be9"
format_stamp: "Formatted at 2026-09-25 16:33:13 on dist-test-slave-65pf"
server_key: "2dc5f509fc48f48659cf7731107be43d"
server_key_iv: "54d004bd8c4bc838e024efaef91fb7c1"
server_key_version: "encryptionkey@0"
I20260925 16:33:13.367913 4565 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 16:33:13.370258 4579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:13.371532 4565 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:33:13.371678 4565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/minicluster-data/ts-0/wal
uuid: "00d60dce358c47209ac0685fd4243be9"
format_stamp: "Formatted at 2026-09-25 16:33:13 on dist-test-slave-65pf"
server_key: "2dc5f509fc48f48659cf7731107be43d"
server_key_iv: "54d004bd8c4bc838e024efaef91fb7c1"
server_key_version: "encryptionkey@0"
I20260925 16:33:13.371770 4565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:33:13.385648 4565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:33:13.388922 4565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:33:13.389073 4565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:33:13.389583 4565 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:33:13.390341 4565 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:33:13.390388 4565 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:13.390426 4565 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:33:13.390441 4565 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:13.399243 4565 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37179
I20260925 16:33:13.399259 4692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37179 every 8 connection(s)
I20260925 16:33:13.399816 4565 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
I20260925 16:33:13.406397 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 4565
I20260925 16:33:13.406528 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/minicluster-data/ts-0/wal/instance
I20260925 16:33:13.406904 392 external_mini_cluster.cc:1428] Setting key 07efdf23d662deac73e55d1b3a51ce17
Sep 25 16:33:13 dist-test-slave-65pf krb5kdc[4459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353993, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:33:13.414076 4542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:13.401461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45821 (local address 127.0.98.62:39347)
0925 16:33:13.401594 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:13.401597 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:13.402414 (+ 817us) server_negotiation.cc:407] Connection header received
0925 16:33:13.404081 (+ 1667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:13.404084 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:13.404140 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:13.404227 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:13.405392 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:13.405959 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:13.407626 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:13.407866 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:13.407980 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:13.410656 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:13.410660 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:13.410662 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:13.410692 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:13.412106 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:13.412626 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:13.412631 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:13.412633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:13.412713 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:13.413039 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:13.413044 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:13.413046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:13.413230 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:13.413334 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:13.413802 (+ 468us) server_negotiation.cc:299] Negotiation successful
0925 16:33:13.413921 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 16:33:13.414975 4695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:13.401807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39347 (local address 127.0.98.1:45821)
0925 16:33:13.402241 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:13.402309 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:13.403874 (+ 1565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:13.404415 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:13.404422 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:13.404771 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:13.405238 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:13.405260 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:13.406089 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:13.406092 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:13.407495 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:13.407505 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:13.408013 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:13.408017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:13.408193 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:13.408817 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:13.408843 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:13.410473 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:33:13.412250 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:13.412256 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:13.412270 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:13.412523 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:13.412806 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:13.412810 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:13.412812 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:13.412946 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:13.413331 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:33:13.413337 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:33:13.413575 (+ 238us) client_negotiation.cc:769] Sending connection context
0925 16:33:13.413765 (+ 190us) client_negotiation.cc:240] Negotiation successful
0925 16:33:13.414072 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":110,"threads_started":1}
I20260925 16:33:13.416182 4693 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39347
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 16:33:13.416446 4693 heartbeater.cc:464] Registering TS with master...
I20260925 16:33:13.416877 4693 heartbeater.cc:511] Master 127.0.98.62:39347 requested a full tablet report, sending...
I20260925 16:33:13.418354 4503 ts_manager.cc:194] Registered new tserver with Master: 00d60dce358c47209ac0685fd4243be9 (127.0.98.1:37179)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260925 16:33:13.419572 4503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45821
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 16:33:13.434185 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39347
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:33:13.548556 4698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:33:13.548837 4698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:33:13.548885 4698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:33:13.553366 4698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:33:13.553434 4698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:33:13.553498 4698 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 16:33:13.559041 4698 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:33:13.560209 4698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:33:13.561257 4698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:33:13.570911 4706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:13.570911 4703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:13.570920 4704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:13.571312 4698 server_base.cc:1034] running on GCE node
I20260925 16:33:13.571753 4698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:33:13.572451 4698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:33:13.573611 4698 hybrid_clock.cc:648] HybridClock initialized: now 1790353993573595 us; error 49 us; skew 500 ppm
Sep 25 16:33:13 dist-test-slave-65pf krb5kdc[4459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353993, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:33:13.576524 4698 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 16:33:13.577278 4698 fs_manager.cc:359] Metadata directory not provided
I20260925 16:33:13.577374 4698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:33:13.577602 4698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:33:13.579751 4698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/minicluster-data/ts-1/data/instance:
uuid: "2a09c0ef5ce24c92960e3f9e6bc35a89"
format_stamp: "Formatted at 2026-09-25 16:33:13 on dist-test-slave-65pf"
server_key: "0c98348108240e0d3b98294bc7977df1"
server_key_iv: "6e8e398da5baf53bbbf331a5a932105b"
server_key_version: "encryptionkey@0"
I20260925 16:33:13.580325 4698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2a09c0ef5ce24c92960e3f9e6bc35a89"
format_stamp: "Formatted at 2026-09-25 16:33:13 on dist-test-slave-65pf"
server_key: "0c98348108240e0d3b98294bc7977df1"
server_key_iv: "6e8e398da5baf53bbbf331a5a932105b"
server_key_version: "encryptionkey@0"
I20260925 16:33:13.583979 4698 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 16:33:13.586351 4712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:13.587685 4698 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:33:13.587818 4698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/minicluster-data/ts-1/wal
uuid: "2a09c0ef5ce24c92960e3f9e6bc35a89"
format_stamp: "Formatted at 2026-09-25 16:33:13 on dist-test-slave-65pf"
server_key: "0c98348108240e0d3b98294bc7977df1"
server_key_iv: "6e8e398da5baf53bbbf331a5a932105b"
server_key_version: "encryptionkey@0"
I20260925 16:33:13.587927 4698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:33:13.602689 4698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:33:13.605957 4698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:33:13.606161 4698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:33:13.606838 4698 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:33:13.607704 4698 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:33:13.607774 4698 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:13.607841 4698 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:33:13.607888 4698 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:13.616472 4698 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45071
I20260925 16:33:13.616492 4825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45071 every 8 connection(s)
I20260925 16:33:13.617067 4698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
I20260925 16:33:13.620693 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 4698
I20260925 16:33:13.620831 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/minicluster-data/ts-1/wal/instance
I20260925 16:33:13.621119 392 external_mini_cluster.cc:1428] Setting key 26b21eab220e242711b20361edbd57db
Sep 25 16:33:13 dist-test-slave-65pf krb5kdc[4459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353993, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260925 16:33:13.631567 4542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:13.618760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46641 (local address 127.0.98.62:39347)
0925 16:33:13.618904 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:13.618912 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:13.619796 (+ 884us) server_negotiation.cc:407] Connection header received
0925 16:33:13.621249 (+ 1453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:13.621252 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:13.621311 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:13.621417 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:13.622834 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:13.623365 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:13.624532 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:13.624762 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:13.624869 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:13.627772 (+ 2903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:13.627775 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:13.627777 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:13.627800 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:13.629290 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:13.629919 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:13.629922 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:13.629923 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:13.629990 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:13.630400 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:13.630403 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:13.630404 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:13.630632 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:13.630734 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:13.631326 (+ 592us) server_negotiation.cc:299] Negotiation successful
0925 16:33:13.631443 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 16:33:13.632388 4828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:13.619164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39347 (local address 127.0.98.2:46641)
0925 16:33:13.619627 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:13.619668 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:13.621034 (+ 1366us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:13.621601 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:13.621608 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:13.622058 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:13.622700 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:13.622717 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:13.623487 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:13.623491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:13.624408 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:13.624418 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:13.624918 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:13.624922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:13.625091 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:13.625693 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:13.625711 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:13.627586 (+ 1875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:33:13.629416 (+ 1830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:13.629425 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:13.629447 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:13.629810 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:13.630077 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:13.630084 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:13.630089 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:13.630294 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:13.630747 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:33:13.630754 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:33:13.630998 (+ 244us) client_negotiation.cc:769] Sending connection context
0925 16:33:13.631258 (+ 260us) client_negotiation.cc:240] Negotiation successful
0925 16:33:13.631526 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":112,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260925 16:33:13.633641 4826 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39347
I20260925 16:33:13.633903 4826 heartbeater.cc:464] Registering TS with master...
I20260925 16:33:13.634429 4826 heartbeater.cc:511] Master 127.0.98.62:39347 requested a full tablet report, sending...
I20260925 16:33:13.635622 4503 ts_manager.cc:194] Registered new tserver with Master: 2a09c0ef5ce24c92960e3f9e6bc35a89 (127.0.98.2:45071)
I20260925 16:33:13.636160 4503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46641
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 16:33:13.649423 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39347
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:33:13.758487 4831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:33:13.758862 4831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:33:13.758948 4831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:33:13.762954 4831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:33:13.763032 4831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:33:13.763120 4831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 16:33:13.768411 4831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:33:13.769438 4831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:33:13.770458 4831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:33:13.779850 4839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:13.779906 4836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:13.780062 4831 server_base.cc:1034] running on GCE node
W20260925 16:33:13.779855 4837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:13.780548 4831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:33:13.781164 4831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:33:13.782342 4831 hybrid_clock.cc:648] HybridClock initialized: now 1790353993782315 us; error 50 us; skew 500 ppm
Sep 25 16:33:13 dist-test-slave-65pf krb5kdc[4459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353993, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:33:13.784987 4831 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 16:33:13.785733 4831 fs_manager.cc:359] Metadata directory not provided
I20260925 16:33:13.785805 4831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:33:13.785969 4831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:33:13.787986 4831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/minicluster-data/ts-2/data/instance:
uuid: "751cd70d2a3d418b99fbf72fd526856e"
format_stamp: "Formatted at 2026-09-25 16:33:13 on dist-test-slave-65pf"
server_key: "8e51505888b0a0ea3947d21c36ec7e4a"
server_key_iv: "3390aa9132aca23b82063d1d8383a6ad"
server_key_version: "encryptionkey@0"
I20260925 16:33:13.788486 4831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/minicluster-data/ts-2/wal/instance:
uuid: "751cd70d2a3d418b99fbf72fd526856e"
format_stamp: "Formatted at 2026-09-25 16:33:13 on dist-test-slave-65pf"
server_key: "8e51505888b0a0ea3947d21c36ec7e4a"
server_key_iv: "3390aa9132aca23b82063d1d8383a6ad"
server_key_version: "encryptionkey@0"
I20260925 16:33:13.792114 4831 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 16:33:13.794440 4845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:13.795706 4831 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:33:13.795815 4831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/minicluster-data/ts-2/wal
uuid: "751cd70d2a3d418b99fbf72fd526856e"
format_stamp: "Formatted at 2026-09-25 16:33:13 on dist-test-slave-65pf"
server_key: "8e51505888b0a0ea3947d21c36ec7e4a"
server_key_iv: "3390aa9132aca23b82063d1d8383a6ad"
server_key_version: "encryptionkey@0"
I20260925 16:33:13.795900 4831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:33:13.811084 4831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:33:13.813884 4831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:33:13.814036 4831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:33:13.814548 4831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:33:13.815471 4831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:33:13.815541 4831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:13.815613 4831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:33:13.815665 4831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:13.824662 4831 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44637
I20260925 16:33:13.824678 4958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44637 every 8 connection(s)
I20260925 16:33:13.825269 4831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
I20260925 16:33:13.825562 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 4831
I20260925 16:33:13.825701 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/minicluster-data/ts-2/wal/instance
I20260925 16:33:13.826053 392 external_mini_cluster.cc:1428] Setting key a47b7a72a29a8ac0136df8361cc65460
Sep 25 16:33:13 dist-test-slave-65pf krb5kdc[4459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353993, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:33:13.838975 4542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:13.827078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39085 (local address 127.0.98.62:39347)
0925 16:33:13.827212 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:13.827217 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:13.828251 (+ 1034us) server_negotiation.cc:407] Connection header received
0925 16:33:13.829593 (+ 1342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:13.829595 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:13.829639 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:13.829705 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:13.830886 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:13.831404 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:13.832596 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:13.832814 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:13.832892 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:13.835419 (+ 2527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:13.835422 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:13.835424 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:13.835447 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:13.836896 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:13.837443 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:13.837445 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:13.837447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:13.837521 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:13.837825 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:13.837827 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:13.837828 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:13.838037 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:13.838114 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:13.838663 (+ 549us) server_negotiation.cc:299] Negotiation successful
0925 16:33:13.838799 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 16:33:13.839926 4961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:13.827622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39347 (local address 127.0.98.3:39085)
0925 16:33:13.828099 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:13.828151 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:13.829415 (+ 1264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:13.829842 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:13.829849 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:13.830194 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:13.830764 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:13.830780 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:13.831550 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:13.831554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:13.832494 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:13.832503 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:13.832939 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:13.832942 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:13.833075 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:13.833706 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:13.833722 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:13.835267 (+ 1545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:33:13.837029 (+ 1762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:13.837034 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:13.837045 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:13.837315 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:13.837599 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:13.837601 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:13.837602 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:13.837746 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:13.838131 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:33:13.838136 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:33:13.838363 (+ 227us) client_negotiation.cc:769] Sending connection context
0925 16:33:13.838566 (+ 203us) client_negotiation.cc:240] Negotiation successful
0925 16:33:13.838983 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":139,"threads_started":1}
I20260925 16:33:13.840960 4959 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39347
I20260925 16:33:13.841210 4959 heartbeater.cc:464] Registering TS with master...
I20260925 16:33:13.841709 4959 heartbeater.cc:511] Master 127.0.98.62:39347 requested a full tablet report, sending...
I20260925 16:33:13.842729 4503 ts_manager.cc:194] Registered new tserver with Master: 751cd70d2a3d418b99fbf72fd526856e (127.0.98.3:44637)
I20260925 16:33:13.843309 4503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39085
I20260925 16:33:13.851941 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 16:33:13.861074 4542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:13.853873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59458 (local address 127.0.98.62:39347)
0925 16:33:13.854001 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:13.854004 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:13.854185 (+ 181us) server_negotiation.cc:407] Connection header received
0925 16:33:13.854281 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:13.854283 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:13.854323 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:13.854390 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:13.855123 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:13.855630 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:13.856412 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:13.856614 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:13.856688 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:13.858026 (+ 1338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:13.858029 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:13.858031 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:13.858052 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:13.859590 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:13.859973 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:13.859976 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:13.859977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:13.860055 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:13.860325 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:13.860327 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:13.860328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:13.860571 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:13.860684 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:13.860850 (+ 166us) server_negotiation.cc:299] Negotiation successful
0925 16:33:13.860945 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 16:33:13.863916 4503 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59458:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 16:33:13.866976 4503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:33:13.881645 4974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:13.876485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34590 (local address 127.0.98.3:44637)
0925 16:33:13.876945 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:13.876959 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:13.876980 (+ 21us) server_negotiation.cc:407] Connection header received
0925 16:33:13.877053 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:13.877061 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:13.877281 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:13.877687 (+ 406us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:33:13.877844 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:13.879076 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:13.880439 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:13.880965 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:13.881094 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:13.881260 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:13.881392 (+ 132us) server_negotiation.cc:299] Negotiation successful
0925 16:33:13.881489 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":143,"threads_started":1}
I20260925 16:33:13.881637 4973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:13.875675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44637 (local address 127.0.0.1:34590)
0925 16:33:13.876471 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:13.876484 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:13.876574 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:13.877425 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:13.877428 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:13.877443 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:33:13.877541 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:13.877547 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:13.879225 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:13.879228 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:13.880328 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:13.880336 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:13.881093 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:13.881096 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:13.881202 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:13.881243 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 16:33:13.881361 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 16:33:13.881472 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":742,"mutex_wait_us":25,"spinlock_wait_cycles":13056,"thread_start_us":120,"threads_started":1}
I20260925 16:33:13.881644 4971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:13.875635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37179 (local address 127.0.0.1:47128)
0925 16:33:13.876058 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:13.876105 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:13.876227 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:13.877236 (+ 1009us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:13.877242 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:13.877267 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:33:13.877425 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:13.877432 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:13.878767 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:13.878773 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:13.880044 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:13.880054 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:13.881139 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:13.881142 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:13.881232 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:13.881256 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 16:33:13.881326 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 16:33:13.881463 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":3,"thread_start_us":87,"threads_started":1}
I20260925 16:33:13.882120 4972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:13.875739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47128 (local address 127.0.98.1:37179)
0925 16:33:13.876726 (+ 987us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:13.876735 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:13.876760 (+ 25us) server_negotiation.cc:407] Connection header received
0925 16:33:13.876839 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:13.876844 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:13.877039 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:13.877585 (+ 546us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:33:13.877745 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:13.878555 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:13.880189 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:13.881010 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:13.881716 (+ 706us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:13.881851 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:13.881898 (+ 47us) server_negotiation.cc:299] Negotiation successful
0925 16:33:13.881980 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":853,"thread_start_us":110,"threads_started":1}
I20260925 16:33:13.885161 4627 tablet_service.cc:1467] Processing CreateTablet for tablet 4d417928aa6d450dbe6d1571014388ea (DEFAULT_TABLE table=test-table [id=ce23bc5692b74b878b43a214d3cee1d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:33:13.885160 4893 tablet_service.cc:1467] Processing CreateTablet for tablet 4d417928aa6d450dbe6d1571014388ea (DEFAULT_TABLE table=test-table [id=ce23bc5692b74b878b43a214d3cee1d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:33:13.886130 4970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:13.876199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45071 (local address 127.0.0.1:49068)
0925 16:33:13.877163 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:13.877176 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:13.877258 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:13.877756 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:13.877759 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:13.877779 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:33:13.877853 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:13.877858 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:13.880450 (+ 2592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:13.880452 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:13.881547 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:13.881556 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:13.882611 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:13.882615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:13.882701 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:13.885829 (+ 3128us) client_negotiation.cc:769] Sending connection context
0925 16:33:13.885911 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 16:33:13.886002 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":911,"thread_start_us":97,"threads_started":1}
I20260925 16:33:13.886366 4893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d417928aa6d450dbe6d1571014388ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:33:13.886360 4627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d417928aa6d450dbe6d1571014388ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:33:13.886638 4975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:13.876532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49068 (local address 127.0.98.2:45071)
0925 16:33:13.876867 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:13.876878 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:13.877355 (+ 477us) server_negotiation.cc:407] Connection header received
0925 16:33:13.877434 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:13.877438 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:13.877617 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:13.877766 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:33:13.877968 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:13.879076 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:13.881677 (+ 2601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:13.882471 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:13.882600 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:13.882724 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:13.886039 (+ 3315us) server_negotiation.cc:299] Negotiation successful
0925 16:33:13.886146 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":135,"threads_started":1}
I20260925 16:33:13.889189 4760 tablet_service.cc:1467] Processing CreateTablet for tablet 4d417928aa6d450dbe6d1571014388ea (DEFAULT_TABLE table=test-table [id=ce23bc5692b74b878b43a214d3cee1d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:33:13.890120 4760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d417928aa6d450dbe6d1571014388ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:33:13.894572 4977 tablet_bootstrap.cc:492] T 4d417928aa6d450dbe6d1571014388ea P 751cd70d2a3d418b99fbf72fd526856e: Bootstrap starting.
I20260925 16:33:13.895015 4976 tablet_bootstrap.cc:492] T 4d417928aa6d450dbe6d1571014388ea P 00d60dce358c47209ac0685fd4243be9: Bootstrap starting.
I20260925 16:33:13.896512 4978 tablet_bootstrap.cc:492] T 4d417928aa6d450dbe6d1571014388ea P 2a09c0ef5ce24c92960e3f9e6bc35a89: Bootstrap starting.
I20260925 16:33:13.897177 4976 tablet_bootstrap.cc:654] T 4d417928aa6d450dbe6d1571014388ea P 00d60dce358c47209ac0685fd4243be9: Neither blocks nor log segments found. Creating new log.
I20260925 16:33:13.898528 4976 log.cc:824] T 4d417928aa6d450dbe6d1571014388ea P 00d60dce358c47209ac0685fd4243be9: Log is configured to *not* fsync() on all Append() calls
I20260925 16:33:13.898769 4978 tablet_bootstrap.cc:654] T 4d417928aa6d450dbe6d1571014388ea P 2a09c0ef5ce24c92960e3f9e6bc35a89: Neither blocks nor log segments found. Creating new log.
I20260925 16:33:13.899221 4977 tablet_bootstrap.cc:654] T 4d417928aa6d450dbe6d1571014388ea P 751cd70d2a3d418b99fbf72fd526856e: Neither blocks nor log segments found. Creating new log.
I20260925 16:33:13.900015 4978 log.cc:824] T 4d417928aa6d450dbe6d1571014388ea P 2a09c0ef5ce24c92960e3f9e6bc35a89: Log is configured to *not* fsync() on all Append() calls
I20260925 16:33:13.900398 4977 log.cc:824] T 4d417928aa6d450dbe6d1571014388ea P 751cd70d2a3d418b99fbf72fd526856e: Log is configured to *not* fsync() on all Append() calls
I20260925 16:33:13.901098 4976 tablet_bootstrap.cc:492] T 4d417928aa6d450dbe6d1571014388ea P 00d60dce358c47209ac0685fd4243be9: No bootstrap required, opened a new log
I20260925 16:33:13.901306 4976 ts_tablet_manager.cc:1397] T 4d417928aa6d450dbe6d1571014388ea P 00d60dce358c47209ac0685fd4243be9: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260925 16:33:13.902031 4978 tablet_bootstrap.cc:492] T 4d417928aa6d450dbe6d1571014388ea P 2a09c0ef5ce24c92960e3f9e6bc35a89: No bootstrap required, opened a new log
I20260925 16:33:13.902213 4978 ts_tablet_manager.cc:1397] T 4d417928aa6d450dbe6d1571014388ea P 2a09c0ef5ce24c92960e3f9e6bc35a89: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260925 16:33:13.902683 4977 tablet_bootstrap.cc:492] T 4d417928aa6d450dbe6d1571014388ea P 751cd70d2a3d418b99fbf72fd526856e: No bootstrap required, opened a new log
I20260925 16:33:13.902865 4977 ts_tablet_manager.cc:1397] T 4d417928aa6d450dbe6d1571014388ea P 751cd70d2a3d418b99fbf72fd526856e: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.002s
I20260925 16:33:13.905329 4976 raft_consensus.cc:348] T 4d417928aa6d450dbe6d1571014388ea P 00d60dce358c47209ac0685fd4243be9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00d60dce358c47209ac0685fd4243be9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37179 } } peers { permanent_uuid: "751cd70d2a3d418b99fbf72fd526856e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44637 } } peers { permanent_uuid: "2a09c0ef5ce24c92960e3f9e6bc35a89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45071 } }
I20260925 16:33:13.905622 4976 raft_consensus.cc:374] T 4d417928aa6d450dbe6d1571014388ea P 00d60dce358c47209ac0685fd4243be9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:33:13.905689 4976 raft_consensus.cc:729] T 4d417928aa6d450dbe6d1571014388ea P 00d60dce358c47209ac0685fd4243be9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00d60dce358c47209ac0685fd4243be9, State: Initialized, Role: FOLLOWER
I20260925 16:33:13.905936 4978 raft_consensus.cc:348] T 4d417928aa6d450dbe6d1571014388ea P 2a09c0ef5ce24c92960e3f9e6bc35a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00d60dce358c47209ac0685fd4243be9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37179 } } peers { permanent_uuid: "751cd70d2a3d418b99fbf72fd526856e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44637 } } peers { permanent_uuid: "2a09c0ef5ce24c92960e3f9e6bc35a89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45071 } }
I20260925 16:33:13.906176 4978 raft_consensus.cc:374] T 4d417928aa6d450dbe6d1571014388ea P 2a09c0ef5ce24c92960e3f9e6bc35a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:33:13.906226 4978 raft_consensus.cc:729] T 4d417928aa6d450dbe6d1571014388ea P 2a09c0ef5ce24c92960e3f9e6bc35a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a09c0ef5ce24c92960e3f9e6bc35a89, State: Initialized, Role: FOLLOWER
I20260925 16:33:13.906042 4977 raft_consensus.cc:348] T 4d417928aa6d450dbe6d1571014388ea P 751cd70d2a3d418b99fbf72fd526856e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00d60dce358c47209ac0685fd4243be9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37179 } } peers { permanent_uuid: "751cd70d2a3d418b99fbf72fd526856e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44637 } } peers { permanent_uuid: "2a09c0ef5ce24c92960e3f9e6bc35a89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45071 } }
I20260925 16:33:13.906144 4976 consensus_queue.cc:260] T 4d417928aa6d450dbe6d1571014388ea P 00d60dce358c47209ac0685fd4243be9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00d60dce358c47209ac0685fd4243be9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37179 } } peers { permanent_uuid: "751cd70d2a3d418b99fbf72fd526856e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44637 } } peers { permanent_uuid: "2a09c0ef5ce24c92960e3f9e6bc35a89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45071 } }
I20260925 16:33:13.906302 4977 raft_consensus.cc:374] T 4d417928aa6d450dbe6d1571014388ea P 751cd70d2a3d418b99fbf72fd526856e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:33:13.906354 4977 raft_consensus.cc:729] T 4d417928aa6d450dbe6d1571014388ea P 751cd70d2a3d418b99fbf72fd526856e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 751cd70d2a3d418b99fbf72fd526856e, State: Initialized, Role: FOLLOWER
I20260925 16:33:13.906697 4978 consensus_queue.cc:260] T 4d417928aa6d450dbe6d1571014388ea P 2a09c0ef5ce24c92960e3f9e6bc35a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00d60dce358c47209ac0685fd4243be9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37179 } } peers { permanent_uuid: "751cd70d2a3d418b99fbf72fd526856e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44637 } } peers { permanent_uuid: "2a09c0ef5ce24c92960e3f9e6bc35a89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45071 } }
I20260925 16:33:13.906804 4977 consensus_queue.cc:260] T 4d417928aa6d450dbe6d1571014388ea P 751cd70d2a3d418b99fbf72fd526856e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00d60dce358c47209ac0685fd4243be9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37179 } } peers { permanent_uuid: "751cd70d2a3d418b99fbf72fd526856e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44637 } } peers { permanent_uuid: "2a09c0ef5ce24c92960e3f9e6bc35a89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45071 } }
I20260925 16:33:13.907276 4976 ts_tablet_manager.cc:1428] T 4d417928aa6d450dbe6d1571014388ea P 00d60dce358c47209ac0685fd4243be9: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260925 16:33:13.907528 4959 heartbeater.cc:503] Master 127.0.98.62:39347 was elected leader, sending a full tablet report...
I20260925 16:33:13.907634 4693 heartbeater.cc:503] Master 127.0.98.62:39347 was elected leader, sending a full tablet report...
I20260925 16:33:13.907773 4977 ts_tablet_manager.cc:1428] T 4d417928aa6d450dbe6d1571014388ea P 751cd70d2a3d418b99fbf72fd526856e: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260925 16:33:13.908494 4978 ts_tablet_manager.cc:1428] T 4d417928aa6d450dbe6d1571014388ea P 2a09c0ef5ce24c92960e3f9e6bc35a89: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 16:33:13.908802 4826 heartbeater.cc:503] Master 127.0.98.62:39347 was elected leader, sending a full tablet report...
W20260925 16:33:14.076252 4960 tablet.cc:2369] T 4d417928aa6d450dbe6d1571014388ea P 751cd70d2a3d418b99fbf72fd526856e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 16:33:14.118127 4827 tablet.cc:2369] T 4d417928aa6d450dbe6d1571014388ea P 2a09c0ef5ce24c92960e3f9e6bc35a89: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 16:33:14.151161 4694 tablet.cc:2369] T 4d417928aa6d450dbe6d1571014388ea P 00d60dce358c47209ac0685fd4243be9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:33:14.193248 4982 raft_consensus.cc:482] T 4d417928aa6d450dbe6d1571014388ea P 00d60dce358c47209ac0685fd4243be9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:33:14.193609 4982 raft_consensus.cc:504] T 4d417928aa6d450dbe6d1571014388ea P 00d60dce358c47209ac0685fd4243be9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00d60dce358c47209ac0685fd4243be9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37179 } } peers { permanent_uuid: "751cd70d2a3d418b99fbf72fd526856e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44637 } } peers { permanent_uuid: "2a09c0ef5ce24c92960e3f9e6bc35a89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45071 } }
I20260925 16:33:14.194864 4982 leader_election.cc:288] T 4d417928aa6d450dbe6d1571014388ea P 00d60dce358c47209ac0685fd4243be9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 751cd70d2a3d418b99fbf72fd526856e (127.0.98.3:44637), 2a09c0ef5ce24c92960e3f9e6bc35a89 (127.0.98.2:45071)
I20260925 16:33:14.200110 4974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:14.195327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40099 (local address 127.0.98.3:44637)
0925 16:33:14.195468 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:14.195472 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:14.195702 (+ 230us) server_negotiation.cc:407] Connection header received
0925 16:33:14.196084 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:14.196086 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:14.196128 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:14.196214 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:33:14.196441 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:14.197173 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:14.198805 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:14.199443 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:14.199614 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:14.199686 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:14.199894 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 16:33:14.199967 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 16:33:14.200110 4975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:14.195313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54059 (local address 127.0.98.2:45071)
0925 16:33:14.195447 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:14.195453 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:14.195669 (+ 216us) server_negotiation.cc:407] Connection header received
0925 16:33:14.195842 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:14.195844 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:14.195890 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:14.195970 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:33:14.196229 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:14.197006 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:14.198800 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:14.199436 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:14.199559 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:14.199634 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:14.199894 (+ 260us) server_negotiation.cc:299] Negotiation successful
0925 16:33:14.199967 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 16:33:14.200188 4985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:14.195190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45071 (local address 127.0.98.1:54059)
0925 16:33:14.195564 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:14.195580 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:14.195738 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:14.195999 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:14.196001 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:14.196016 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:33:14.196117 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:14.196122 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:14.197127 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:14.197129 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:14.198667 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:14.198678 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:14.199579 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:14.199583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:14.199731 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:14.199784 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 16:33:14.199860 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 16:33:14.200010 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":138,"spinlock_wait_cycles":24576,"thread_start_us":103,"threads_started":1}
I20260925 16:33:14.200260 4986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:14.195198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44637 (local address 127.0.98.1:40099)
0925 16:33:14.195565 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:14.195580 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:14.195738 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:14.196237 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:14.196240 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:14.196259 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:33:14.196341 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:14.196346 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:14.197269 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:14.197272 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:14.198667 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:14.198678 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:14.199579 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:14.199583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:14.199735 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:14.199779 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 16:33:14.199860 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 16:33:14.200011 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"mutex_wait_us":237,"spinlock_wait_cycles":49664,"thread_start_us":126,"threads_started":1}
I20260925 16:33:14.201000 4780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d417928aa6d450dbe6d1571014388ea" candidate_uuid: "00d60dce358c47209ac0685fd4243be9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a09c0ef5ce24c92960e3f9e6bc35a89" is_pre_election: true
I20260925 16:33:14.201319 4780 raft_consensus.cc:2454] T 4d417928aa6d450dbe6d1571014388ea P 2a09c0ef5ce24c92960e3f9e6bc35a89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00d60dce358c47209ac0685fd4243be9 in term 0.
I20260925 16:33:14.201310 4913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d417928aa6d450dbe6d1571014388ea" candidate_uuid: "00d60dce358c47209ac0685fd4243be9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "751cd70d2a3d418b99fbf72fd526856e" is_pre_election: true
I20260925 16:33:14.201620 4913 raft_consensus.cc:2454] T 4d417928aa6d450dbe6d1571014388ea P 751cd70d2a3d418b99fbf72fd526856e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00d60dce358c47209ac0685fd4243be9 in term 0.
I20260925 16:33:14.201853 4580 leader_election.cc:302] T 4d417928aa6d450dbe6d1571014388ea P 00d60dce358c47209ac0685fd4243be9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00d60dce358c47209ac0685fd4243be9, 2a09c0ef5ce24c92960e3f9e6bc35a89; no voters:
I20260925 16:33:14.202131 4982 raft_consensus.cc:2781] T 4d417928aa6d450dbe6d1571014388ea P 00d60dce358c47209ac0685fd4243be9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:33:14.202226 4982 raft_consensus.cc:482] T 4d417928aa6d450dbe6d1571014388ea P 00d60dce358c47209ac0685fd4243be9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:33:14.202291 4982 raft_consensus.cc:3037] T 4d417928aa6d450dbe6d1571014388ea P 00d60dce358c47209ac0685fd4243be9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:33:14.203419 4982 raft_consensus.cc:504] T 4d417928aa6d450dbe6d1571014388ea P 00d60dce358c47209ac0685fd4243be9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00d60dce358c47209ac0685fd4243be9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37179 } } peers { permanent_uuid: "751cd70d2a3d418b99fbf72fd526856e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44637 } } peers { permanent_uuid: "2a09c0ef5ce24c92960e3f9e6bc35a89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45071 } }
I20260925 16:33:14.203857 4982 leader_election.cc:288] T 4d417928aa6d450dbe6d1571014388ea P 00d60dce358c47209ac0685fd4243be9 [CANDIDATE]: Term 1 election: Requested vote from peers 751cd70d2a3d418b99fbf72fd526856e (127.0.98.3:44637), 2a09c0ef5ce24c92960e3f9e6bc35a89 (127.0.98.2:45071)
I20260925 16:33:14.204272 4913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d417928aa6d450dbe6d1571014388ea" candidate_uuid: "00d60dce358c47209ac0685fd4243be9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "751cd70d2a3d418b99fbf72fd526856e"
I20260925 16:33:14.204315 4780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d417928aa6d450dbe6d1571014388ea" candidate_uuid: "00d60dce358c47209ac0685fd4243be9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a09c0ef5ce24c92960e3f9e6bc35a89"
I20260925 16:33:14.204422 4913 raft_consensus.cc:3037] T 4d417928aa6d450dbe6d1571014388ea P 751cd70d2a3d418b99fbf72fd526856e [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:33:14.204438 4780 raft_consensus.cc:3037] T 4d417928aa6d450dbe6d1571014388ea P 2a09c0ef5ce24c92960e3f9e6bc35a89 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:33:14.205487 4780 raft_consensus.cc:2454] T 4d417928aa6d450dbe6d1571014388ea P 2a09c0ef5ce24c92960e3f9e6bc35a89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00d60dce358c47209ac0685fd4243be9 in term 1.
I20260925 16:33:14.205801 4580 leader_election.cc:302] T 4d417928aa6d450dbe6d1571014388ea P 00d60dce358c47209ac0685fd4243be9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00d60dce358c47209ac0685fd4243be9, 2a09c0ef5ce24c92960e3f9e6bc35a89; no voters:
I20260925 16:33:14.205801 4913 raft_consensus.cc:2454] T 4d417928aa6d450dbe6d1571014388ea P 751cd70d2a3d418b99fbf72fd526856e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00d60dce358c47209ac0685fd4243be9 in term 1.
I20260925 16:33:14.205987 4982 raft_consensus.cc:2781] T 4d417928aa6d450dbe6d1571014388ea P 00d60dce358c47209ac0685fd4243be9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:33:14.206218 4982 raft_consensus.cc:686] T 4d417928aa6d450dbe6d1571014388ea P 00d60dce358c47209ac0685fd4243be9 [term 1 LEADER]: Becoming Leader. State: Replica: 00d60dce358c47209ac0685fd4243be9, State: Running, Role: LEADER
I20260925 16:33:14.206573 4982 consensus_queue.cc:237] T 4d417928aa6d450dbe6d1571014388ea P 00d60dce358c47209ac0685fd4243be9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00d60dce358c47209ac0685fd4243be9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37179 } } peers { permanent_uuid: "751cd70d2a3d418b99fbf72fd526856e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44637 } } peers { permanent_uuid: "2a09c0ef5ce24c92960e3f9e6bc35a89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45071 } }
I20260925 16:33:14.209833 4502 catalog_manager.cc:5495] T 4d417928aa6d450dbe6d1571014388ea P 00d60dce358c47209ac0685fd4243be9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00d60dce358c47209ac0685fd4243be9 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "00d60dce358c47209ac0685fd4243be9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00d60dce358c47209ac0685fd4243be9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "751cd70d2a3d418b99fbf72fd526856e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a09c0ef5ce24c92960e3f9e6bc35a89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45071 } health_report { overall_health: UNKNOWN } } }
I20260925 16:33:14.245046 4972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:14.241583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47136 (local address 127.0.98.1:37179)
0925 16:33:14.241713 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:14.241717 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:14.241730 (+ 13us) server_negotiation.cc:407] Connection header received
0925 16:33:14.241859 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:14.241861 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:14.241916 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:14.242002 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:33:14.242281 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:14.242843 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:14.243655 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:14.243858 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:14.243929 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:14.244191 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:33:14.244640 (+ 449us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:33:14.244760 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:14.244817 (+ 57us) server_negotiation.cc:299] Negotiation successful
0925 16:33:14.244866 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 16:33:14.251006 4780 raft_consensus.cc:1261] T 4d417928aa6d450dbe6d1571014388ea P 2a09c0ef5ce24c92960e3f9e6bc35a89 [term 1 FOLLOWER]: Refusing update from remote peer 00d60dce358c47209ac0685fd4243be9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:33:14.251006 4913 raft_consensus.cc:1261] T 4d417928aa6d450dbe6d1571014388ea P 751cd70d2a3d418b99fbf72fd526856e [term 1 FOLLOWER]: Refusing update from remote peer 00d60dce358c47209ac0685fd4243be9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:33:14.251737 4987 consensus_queue.cc:1035] T 4d417928aa6d450dbe6d1571014388ea P 00d60dce358c47209ac0685fd4243be9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "751cd70d2a3d418b99fbf72fd526856e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:33:14.251940 4982 consensus_queue.cc:1035] T 4d417928aa6d450dbe6d1571014388ea P 00d60dce358c47209ac0685fd4243be9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a09c0ef5ce24c92960e3f9e6bc35a89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:33:14.259972 4992 mvcc.cc:205] Tried to move back new op lower bound from 7333289960444583936 to 7333289960273186816. Current Snapshot: MvccSnapshot[applied={T|T < 7333289960444583936}]
I20260925 16:33:14.269793 4502 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:33:14.270016 4502 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:33:14.272526 4502 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 166ff127944b437baee01105a5b9094a: Sending DeleteTablet for 3 replicas of tablet 4d417928aa6d450dbe6d1571014388ea
I20260925 16:33:14.273322 4627 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d417928aa6d450dbe6d1571014388ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:33:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47128
I20260925 16:33:14.273391 4760 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d417928aa6d450dbe6d1571014388ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:33:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49068
I20260925 16:33:14.273478 4893 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d417928aa6d450dbe6d1571014388ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:33:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34590
I20260925 16:33:14.273861 5001 tablet_replica.cc:330] T 4d417928aa6d450dbe6d1571014388ea P 2a09c0ef5ce24c92960e3f9e6bc35a89: stopping tablet replica
I20260925 16:33:14.274047 5002 tablet_replica.cc:330] T 4d417928aa6d450dbe6d1571014388ea P 751cd70d2a3d418b99fbf72fd526856e: stopping tablet replica
I20260925 16:33:14.274361 5002 raft_consensus.cc:2229] T 4d417928aa6d450dbe6d1571014388ea P 751cd70d2a3d418b99fbf72fd526856e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:33:14.274536 5001 raft_consensus.cc:2229] T 4d417928aa6d450dbe6d1571014388ea P 2a09c0ef5ce24c92960e3f9e6bc35a89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:33:14.274641 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 4565
I20260925 16:33:14.274722 5002 raft_consensus.cc:2258] T 4d417928aa6d450dbe6d1571014388ea P 751cd70d2a3d418b99fbf72fd526856e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:33:14.276098 5001 raft_consensus.cc:2258] T 4d417928aa6d450dbe6d1571014388ea P 2a09c0ef5ce24c92960e3f9e6bc35a89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:33:14.277235 5002 ts_tablet_manager.cc:1905] T 4d417928aa6d450dbe6d1571014388ea P 751cd70d2a3d418b99fbf72fd526856e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:33:14.277580 5001 ts_tablet_manager.cc:1905] T 4d417928aa6d450dbe6d1571014388ea P 2a09c0ef5ce24c92960e3f9e6bc35a89: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:33:14.279592 5002 ts_tablet_manager.cc:1918] T 4d417928aa6d450dbe6d1571014388ea P 751cd70d2a3d418b99fbf72fd526856e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:33:14.279703 5002 log.cc:1196] T 4d417928aa6d450dbe6d1571014388ea P 751cd70d2a3d418b99fbf72fd526856e: Deleting WAL directory at /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/minicluster-data/ts-2/wal/wals/4d417928aa6d450dbe6d1571014388ea
I20260925 16:33:14.280045 5002 ts_tablet_manager.cc:1939] T 4d417928aa6d450dbe6d1571014388ea P 751cd70d2a3d418b99fbf72fd526856e: Deleting consensus metadata
I20260925 16:33:14.280349 5001 ts_tablet_manager.cc:1918] T 4d417928aa6d450dbe6d1571014388ea P 2a09c0ef5ce24c92960e3f9e6bc35a89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:33:14.280447 5001 log.cc:1196] T 4d417928aa6d450dbe6d1571014388ea P 2a09c0ef5ce24c92960e3f9e6bc35a89: Deleting WAL directory at /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353966814071-392-0/minicluster-data/ts-1/wal/wals/4d417928aa6d450dbe6d1571014388ea
I20260925 16:33:14.280764 5001 ts_tablet_manager.cc:1939] T 4d417928aa6d450dbe6d1571014388ea P 2a09c0ef5ce24c92960e3f9e6bc35a89: Deleting consensus metadata
I20260925 16:33:14.281355 4490 catalog_manager.cc:4841] TS 751cd70d2a3d418b99fbf72fd526856e (127.0.98.3:44637): tablet 4d417928aa6d450dbe6d1571014388ea (table test-table [id=ce23bc5692b74b878b43a214d3cee1d3]) successfully deleted
I20260925 16:33:14.282027 4487 catalog_manager.cc:4841] TS 2a09c0ef5ce24c92960e3f9e6bc35a89 (127.0.98.2:45071): tablet 4d417928aa6d450dbe6d1571014388ea (table test-table [id=ce23bc5692b74b878b43a214d3cee1d3]) successfully deleted
W20260925 16:33:14.282466 4489 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37179 (error 108)
I20260925 16:33:14.282732 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 4698
I20260925 16:33:14.283916 4970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:14.283578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37179 (local address 127.0.0.1:47144)
0925 16:33:14.283731 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:14.283815 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 16:33:14.284173 4489 catalog_manager.cc:4584] TS 00d60dce358c47209ac0685fd4243be9 (127.0.98.1:37179): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4d417928aa6d450dbe6d1571014388ea: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37179: connect: Connection refused (error 111)
I20260925 16:33:14.288508 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 4831
I20260925 16:33:14.293936 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 4472
2026-09-25T16:33:14Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2545 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-25T16:33:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:33:14Z Disabled control of system clock
I20260925 16:33:14.323736 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353966814071-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353966814071-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353966814071-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41271
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41271
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 16:33:14.441141 5007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:33:14.441419 5007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:33:14.441455 5007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:33:14.445397 5007 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:33:14.445457 5007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:33:14.445472 5007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:33:14.445489 5007 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:33:14.445504 5007 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:33:14.450349 5007 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353966814071-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353966814071-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41271
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353966814071-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:33:14.451525 5007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:33:14.452884 5007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 16:33:14.461715 5007 server_base.cc:1034] running on GCE node
W20260925 16:33:14.461658 5013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:14.461654 5015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:14.461665 5012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:14.462559 5007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:33:14.463734 5007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:33:14.464932 5007 hybrid_clock.cc:648] HybridClock initialized: now 1790353994464922 us; error 31 us; skew 500 ppm
I20260925 16:33:14.466955 5007 webserver.cc:460] Webserver started at http://127.0.98.62:34657/ using document root <none> and password file <none>
I20260925 16:33:14.467586 5007 fs_manager.cc:359] Metadata directory not provided
I20260925 16:33:14.467680 5007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:33:14.467913 5007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:33:14.469950 5007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353966814071-392-0/minicluster-data/master-0/data/instance:
uuid: "812db122f8e04cfeace7cd6fbe689252"
format_stamp: "Formatted at 2026-09-25 16:33:14 on dist-test-slave-65pf"
server_key: "e883af0b8efeaa1afc545635792042aa"
server_key_iv: "7b2b4285286af5c563af2bdd0b0edd17"
server_key_version: "encryptionkey@0"
I20260925 16:33:14.470546 5007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353966814071-392-0/minicluster-data/master-0/wal/instance:
uuid: "812db122f8e04cfeace7cd6fbe689252"
format_stamp: "Formatted at 2026-09-25 16:33:14 on dist-test-slave-65pf"
server_key: "e883af0b8efeaa1afc545635792042aa"
server_key_iv: "7b2b4285286af5c563af2bdd0b0edd17"
server_key_version: "encryptionkey@0"
I20260925 16:33:14.474370 5007 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 16:33:14.476948 5021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:14.478302 5007 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:33:14.478474 5007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353966814071-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353966814071-392-0/minicluster-data/master-0/wal
uuid: "812db122f8e04cfeace7cd6fbe689252"
format_stamp: "Formatted at 2026-09-25 16:33:14 on dist-test-slave-65pf"
server_key: "e883af0b8efeaa1afc545635792042aa"
server_key_iv: "7b2b4285286af5c563af2bdd0b0edd17"
server_key_version: "encryptionkey@0"
I20260925 16:33:14.478643 5007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353966814071-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353966814071-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353966814071-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:33:14.483495 5007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:33:14.484160 5007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:33:14.484372 5007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:33:14.491460 5007 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41271
I20260925 16:33:14.491479 5073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41271 every 8 connection(s)
I20260925 16:33:14.492473 5007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:33:14.495743 5074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:33:14.499480 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 5007
I20260925 16:33:14.499593 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353966814071-392-0/minicluster-data/master-0/wal/instance
I20260925 16:33:14.499913 392 external_mini_cluster.cc:1428] Setting key c2a98521a4d48030d67e7c1f530a6880
I20260925 16:33:14.501724 5074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 812db122f8e04cfeace7cd6fbe689252: Bootstrap starting.
I20260925 16:33:14.504508 5074 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 812db122f8e04cfeace7cd6fbe689252: Neither blocks nor log segments found. Creating new log.
I20260925 16:33:14.505465 5074 log.cc:824] T 00000000000000000000000000000000 P 812db122f8e04cfeace7cd6fbe689252: Log is configured to *not* fsync() on all Append() calls
I20260925 16:33:14.506382 5077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:14.501444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60418 (local address 127.0.98.62:41271)
0925 16:33:14.502161 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:14.502181 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:14.502236 (+ 55us) server_negotiation.cc:407] Connection header received
0925 16:33:14.503157 (+ 921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:14.503182 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:14.503642 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:14.504020 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:14.504873 (+ 853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:14.504879 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:14.504895 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:33:14.504907 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:14.504978 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:33:14.505001 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:14.505111 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:14.505250 (+ 139us) server_negotiation.cc:299] Negotiation successful
0925 16:33:14.505502 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":109,"threads_started":1}
I20260925 16:33:14.507689 5074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 812db122f8e04cfeace7cd6fbe689252: No bootstrap required, opened a new log
I20260925 16:33:14.511409 5074 raft_consensus.cc:348] T 00000000000000000000000000000000 P 812db122f8e04cfeace7cd6fbe689252 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "812db122f8e04cfeace7cd6fbe689252" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41271 } }
I20260925 16:33:14.511705 5074 raft_consensus.cc:374] T 00000000000000000000000000000000 P 812db122f8e04cfeace7cd6fbe689252 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:33:14.511791 5074 raft_consensus.cc:729] T 00000000000000000000000000000000 P 812db122f8e04cfeace7cd6fbe689252 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 812db122f8e04cfeace7cd6fbe689252, State: Initialized, Role: FOLLOWER
I20260925 16:33:14.512260 5074 consensus_queue.cc:260] T 00000000000000000000000000000000 P 812db122f8e04cfeace7cd6fbe689252 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "812db122f8e04cfeace7cd6fbe689252" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41271 } }
I20260925 16:33:14.512392 5074 raft_consensus.cc:388] T 00000000000000000000000000000000 P 812db122f8e04cfeace7cd6fbe689252 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:33:14.512434 5074 raft_consensus.cc:482] T 00000000000000000000000000000000 P 812db122f8e04cfeace7cd6fbe689252 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:33:14.512509 5074 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 812db122f8e04cfeace7cd6fbe689252 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:33:14.513496 5074 raft_consensus.cc:504] T 00000000000000000000000000000000 P 812db122f8e04cfeace7cd6fbe689252 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "812db122f8e04cfeace7cd6fbe689252" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41271 } }
I20260925 16:33:14.513814 5074 leader_election.cc:302] T 00000000000000000000000000000000 P 812db122f8e04cfeace7cd6fbe689252 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 812db122f8e04cfeace7cd6fbe689252; no voters:
I20260925 16:33:14.514082 5074 leader_election.cc:288] T 00000000000000000000000000000000 P 812db122f8e04cfeace7cd6fbe689252 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:33:14.514271 5079 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 812db122f8e04cfeace7cd6fbe689252 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:33:14.514509 5079 raft_consensus.cc:686] T 00000000000000000000000000000000 P 812db122f8e04cfeace7cd6fbe689252 [term 1 LEADER]: Becoming Leader. State: Replica: 812db122f8e04cfeace7cd6fbe689252, State: Running, Role: LEADER
I20260925 16:33:14.514871 5079 consensus_queue.cc:237] T 00000000000000000000000000000000 P 812db122f8e04cfeace7cd6fbe689252 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "812db122f8e04cfeace7cd6fbe689252" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41271 } }
I20260925 16:33:14.515480 5074 sys_catalog.cc:564] T 00000000000000000000000000000000 P 812db122f8e04cfeace7cd6fbe689252 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:33:14.516589 5080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 812db122f8e04cfeace7cd6fbe689252 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "812db122f8e04cfeace7cd6fbe689252" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "812db122f8e04cfeace7cd6fbe689252" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41271 } } }
I20260925 16:33:14.516723 5080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 812db122f8e04cfeace7cd6fbe689252 [sys.catalog]: This master's current role is: LEADER
I20260925 16:33:14.516921 5081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 812db122f8e04cfeace7cd6fbe689252 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 812db122f8e04cfeace7cd6fbe689252. Latest consensus state: current_term: 1 leader_uuid: "812db122f8e04cfeace7cd6fbe689252" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "812db122f8e04cfeace7cd6fbe689252" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41271 } } }
I20260925 16:33:14.517072 5081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 812db122f8e04cfeace7cd6fbe689252 [sys.catalog]: This master's current role is: LEADER
I20260925 16:33:14.517125 5088 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:33:14.520341 5088 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:33:14.526198 5088 catalog_manager.cc:1347] Generated new cluster ID: 6208fcfda09b40beaf79a56daab67414
I20260925 16:33:14.526298 5088 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:33:14.537168 5088 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:33:14.537535 5088 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:33:14.544070 5088 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 812db122f8e04cfeace7cd6fbe689252: Generated new TSK 0
I20260925 16:33:14.544972 5088 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:33:14.555241 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353966814071-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353966814071-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353966814071-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41271
--builtin_ntp_servers=127.0.98.20:38421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 16:33:14.681988 5098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:33:14.682245 5098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:33:14.682286 5098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:33:14.686333 5098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:33:14.686390 5098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:33:14.686445 5098 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 16:33:14.691135 5098 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353966814071-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353966814071-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353966814071-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:33:14.692237 5098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:33:14.693578 5098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:33:14.703156 5104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:14.703131 5106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:14.703121 5103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:14.703361 5098 server_base.cc:1034] running on GCE node
I20260925 16:33:14.703863 5098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:33:14.704466 5098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:33:14.705629 5098 hybrid_clock.cc:648] HybridClock initialized: now 1790353994705615 us; error 26 us; skew 500 ppm
I20260925 16:33:14.707464 5098 webserver.cc:460] Webserver started at http://127.0.98.1:33769/ using document root <none> and password file <none>
I20260925 16:33:14.708000 5098 fs_manager.cc:359] Metadata directory not provided
I20260925 16:33:14.708060 5098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:33:14.708230 5098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:33:14.710229 5098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353966814071-392-0/minicluster-data/ts-0/data/instance:
uuid: "afa8120b5a8e4b3d9d311028d23b90f1"
format_stamp: "Formatted at 2026-09-25 16:33:14 on dist-test-slave-65pf"
server_key: "8fdec58c0a001fd128c31478987b3dbb"
server_key_iv: "77bc8dafa489bd8a3b55f6b101fdfb4d"
server_key_version: "encryptionkey@0"
I20260925 16:33:14.710795 5098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353966814071-392-0/minicluster-data/ts-0/wal/instance:
uuid: "afa8120b5a8e4b3d9d311028d23b90f1"
format_stamp: "Formatted at 2026-09-25 16:33:14 on dist-test-slave-65pf"
server_key: "8fdec58c0a001fd128c31478987b3dbb"
server_key_iv: "77bc8dafa489bd8a3b55f6b101fdfb4d"
server_key_version: "encryptionkey@0"
I20260925 16:33:14.714268 5098 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 16:33:14.716897 5112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:14.718153 5098 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 16:33:14.718292 5098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353966814071-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353966814071-392-0/minicluster-data/ts-0/wal
uuid: "afa8120b5a8e4b3d9d311028d23b90f1"
format_stamp: "Formatted at 2026-09-25 16:33:14 on dist-test-slave-65pf"
server_key: "8fdec58c0a001fd128c31478987b3dbb"
server_key_iv: "77bc8dafa489bd8a3b55f6b101fdfb4d"
server_key_version: "encryptionkey@0"
I20260925 16:33:14.718400 5098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353966814071-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353966814071-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353966814071-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:33:14.723474 5098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:33:14.724073 5098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:33:14.724265 5098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:33:14.724810 5098 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:33:14.725703 5098 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:33:14.725751 5098 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:14.725818 5098 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:33:14.725849 5098 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:14.734438 5098 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37225
I20260925 16:33:14.734459 5225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37225 every 8 connection(s)
I20260925 16:33:14.735400 5098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
I20260925 16:33:14.741627 5077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:14.737306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33399 (local address 127.0.98.62:41271)
0925 16:33:14.737441 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:14.737444 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:14.738499 (+ 1055us) server_negotiation.cc:407] Connection header received
0925 16:33:14.739861 (+ 1362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:14.739865 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:14.739908 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:14.740016 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:14.741161 (+ 1145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:14.741165 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:14.741167 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:33:14.741180 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:14.741206 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:33:14.741215 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:14.741315 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:14.741448 (+ 133us) server_negotiation.cc:299] Negotiation successful
0925 16:33:14.741493 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 16:33:14.742336 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 5098
I20260925 16:33:14.742472 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353966814071-392-0/minicluster-data/ts-0/wal/instance
I20260925 16:33:14.742621 5228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:14.737725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41271 (local address 127.0.98.1:33399)
0925 16:33:14.738354 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:14.738393 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:14.739636 (+ 1243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:14.740179 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:14.740185 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:14.740273 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:14.740882 (+ 609us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 16:33:14.740884 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:14.740923 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 16:33:14.740989 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:33:14.741297 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:33:14.741306 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:33:14.741311 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 16:33:14.741491 (+ 180us) client_negotiation.cc:240] Negotiation successful
0925 16:33:14.741732 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":108,"threads_started":1}
I20260925 16:33:14.742813 392 external_mini_cluster.cc:1428] Setting key a5f4efa6202a35fb02e93e52b2511791
I20260925 16:33:14.743834 5226 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41271
I20260925 16:33:14.744081 5226 heartbeater.cc:464] Registering TS with master...
I20260925 16:33:14.744537 5226 heartbeater.cc:511] Master 127.0.98.62:41271 requested a full tablet report, sending...
I20260925 16:33:14.744761 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353966814071-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353966814071-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353966814071-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41271
--builtin_ntp_servers=127.0.98.20:38421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 16:33:14.746085 5038 ts_manager.cc:194] Registered new tserver with Master: afa8120b5a8e4b3d9d311028d23b90f1 (127.0.98.1:37225)
W20260925 16:33:14.858180 5229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:33:14.858448 5229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:33:14.858536 5229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:33:14.863095 5229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:33:14.863164 5229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:33:14.863272 5229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 16:33:14.868276 5229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353966814071-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353966814071-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353966814071-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:33:14.869410 5229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:33:14.870952 5229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:33:14.880533 5234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:14.880641 5229 server_base.cc:1034] running on GCE node
W20260925 16:33:14.880520 5237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:14.880520 5235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:14.881268 5229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:33:14.881954 5229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:33:14.883111 5229 hybrid_clock.cc:648] HybridClock initialized: now 1790353994883091 us; error 33 us; skew 500 ppm
I20260925 16:33:14.885040 5229 webserver.cc:460] Webserver started at http://127.0.98.2:42815/ using document root <none> and password file <none>
I20260925 16:33:14.885650 5229 fs_manager.cc:359] Metadata directory not provided
I20260925 16:33:14.885713 5229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:33:14.885926 5229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:33:14.888046 5229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353966814071-392-0/minicluster-data/ts-1/data/instance:
uuid: "1b38e884ed684c4b97742ff7b815fb75"
format_stamp: "Formatted at 2026-09-25 16:33:14 on dist-test-slave-65pf"
server_key: "87ec4bd39c4fa36a24ba99f1d2aa1c81"
server_key_iv: "c1f024fd91ee46e8bf1360f14c80271e"
server_key_version: "encryptionkey@0"
I20260925 16:33:14.888599 5229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353966814071-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1b38e884ed684c4b97742ff7b815fb75"
format_stamp: "Formatted at 2026-09-25 16:33:14 on dist-test-slave-65pf"
server_key: "87ec4bd39c4fa36a24ba99f1d2aa1c81"
server_key_iv: "c1f024fd91ee46e8bf1360f14c80271e"
server_key_version: "encryptionkey@0"
I20260925 16:33:14.892414 5229 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 16:33:14.895231 5243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:14.896585 5229 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 16:33:14.896754 5229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353966814071-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353966814071-392-0/minicluster-data/ts-1/wal
uuid: "1b38e884ed684c4b97742ff7b815fb75"
format_stamp: "Formatted at 2026-09-25 16:33:14 on dist-test-slave-65pf"
server_key: "87ec4bd39c4fa36a24ba99f1d2aa1c81"
server_key_iv: "c1f024fd91ee46e8bf1360f14c80271e"
server_key_version: "encryptionkey@0"
I20260925 16:33:14.896883 5229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353966814071-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353966814071-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353966814071-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:33:14.901654 5229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:33:14.902297 5229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:33:14.902508 5229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:33:14.903105 5229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:33:14.904066 5229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:33:14.904143 5229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:14.904212 5229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:33:14.904270 5229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:14.913911 5229 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46003
I20260925 16:33:14.913931 5356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46003 every 8 connection(s)
I20260925 16:33:14.914975 5229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
I20260925 16:33:14.921301 5077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:14.916964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34985 (local address 127.0.98.62:41271)
0925 16:33:14.917103 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:14.917106 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:14.918187 (+ 1081us) server_negotiation.cc:407] Connection header received
0925 16:33:14.919552 (+ 1365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:14.919555 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:14.919599 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:14.919712 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:14.920846 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:14.920849 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:14.920851 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:33:14.920864 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:14.920883 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:33:14.920898 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:14.920988 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:14.921127 (+ 139us) server_negotiation.cc:299] Negotiation successful
0925 16:33:14.921171 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 16:33:14.921344 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 5229
I20260925 16:33:14.921506 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353966814071-392-0/minicluster-data/ts-1/wal/instance
I20260925 16:33:14.921854 392 external_mini_cluster.cc:1428] Setting key adc661f9b66589400e90b3dbf88036ab
I20260925 16:33:14.922319 5359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:14.917415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41271 (local address 127.0.98.2:34985)
0925 16:33:14.918036 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:14.918075 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:14.919344 (+ 1269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:14.919876 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:14.919883 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:14.919968 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:14.920597 (+ 629us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 16:33:14.920599 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:14.920636 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 16:33:14.920702 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:33:14.920981 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:33:14.920989 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:33:14.920996 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 16:33:14.921175 (+ 179us) client_negotiation.cc:240] Negotiation successful
0925 16:33:14.921420 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":103,"threads_started":1}
I20260925 16:33:14.923448 5357 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41271
I20260925 16:33:14.923683 5357 heartbeater.cc:464] Registering TS with master...
I20260925 16:33:14.924028 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353966814071-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353966814071-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353966814071-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41271
--builtin_ntp_servers=127.0.98.20:38421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 16:33:14.924171 5357 heartbeater.cc:511] Master 127.0.98.62:41271 requested a full tablet report, sending...
I20260925 16:33:14.925271 5038 ts_manager.cc:194] Registered new tserver with Master: 1b38e884ed684c4b97742ff7b815fb75 (127.0.98.2:46003)
W20260925 16:33:15.035326 5360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:33:15.035596 5360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:33:15.035638 5360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:33:15.039806 5360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:33:15.039863 5360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:33:15.039942 5360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 16:33:15.044862 5360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353966814071-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353966814071-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353966814071-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:33:15.046013 5360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:33:15.047461 5360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:33:15.056951 5365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:15.056944 5368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:15.056941 5366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:15.057050 5360 server_base.cc:1034] running on GCE node
I20260925 16:33:15.057554 5360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:33:15.058195 5360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:33:15.059394 5360 hybrid_clock.cc:648] HybridClock initialized: now 1790353995059377 us; error 36 us; skew 500 ppm
I20260925 16:33:15.061321 5360 webserver.cc:460] Webserver started at http://127.0.98.3:44435/ using document root <none> and password file <none>
I20260925 16:33:15.061901 5360 fs_manager.cc:359] Metadata directory not provided
I20260925 16:33:15.061996 5360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:33:15.062240 5360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:33:15.064327 5360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353966814071-392-0/minicluster-data/ts-2/data/instance:
uuid: "5f6fa48b06a84617ae2823a9e4ccadf0"
format_stamp: "Formatted at 2026-09-25 16:33:15 on dist-test-slave-65pf"
server_key: "7602232977e94a4c1c32aaebaeb50cb0"
server_key_iv: "dcb002938a4cba447c6c7b43cdc5078c"
server_key_version: "encryptionkey@0"
I20260925 16:33:15.064898 5360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353966814071-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5f6fa48b06a84617ae2823a9e4ccadf0"
format_stamp: "Formatted at 2026-09-25 16:33:15 on dist-test-slave-65pf"
server_key: "7602232977e94a4c1c32aaebaeb50cb0"
server_key_iv: "dcb002938a4cba447c6c7b43cdc5078c"
server_key_version: "encryptionkey@0"
I20260925 16:33:15.068602 5360 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 16:33:15.071199 5374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:15.072389 5360 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:33:15.072535 5360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353966814071-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353966814071-392-0/minicluster-data/ts-2/wal
uuid: "5f6fa48b06a84617ae2823a9e4ccadf0"
format_stamp: "Formatted at 2026-09-25 16:33:15 on dist-test-slave-65pf"
server_key: "7602232977e94a4c1c32aaebaeb50cb0"
server_key_iv: "dcb002938a4cba447c6c7b43cdc5078c"
server_key_version: "encryptionkey@0"
I20260925 16:33:15.072664 5360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353966814071-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353966814071-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353966814071-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:33:15.077313 5360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:33:15.077917 5360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:33:15.078099 5360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:33:15.078662 5360 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:33:15.079533 5360 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:33:15.079603 5360 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:15.079690 5360 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:33:15.079741 5360 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:15.088408 5360 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34127
I20260925 16:33:15.088478 5487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34127 every 8 connection(s)
I20260925 16:33:15.089349 5360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
I20260925 16:33:15.090365 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 5360
I20260925 16:33:15.090477 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353966814071-392-0/minicluster-data/ts-2/wal/instance
I20260925 16:33:15.090790 392 external_mini_cluster.cc:1428] Setting key 5c2809035dc36066361880c1849f269a
I20260925 16:33:15.095903 5077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:15.091408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40453 (local address 127.0.98.62:41271)
0925 16:33:15.091549 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:15.091553 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:15.092831 (+ 1278us) server_negotiation.cc:407] Connection header received
0925 16:33:15.094188 (+ 1357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:15.094191 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:15.094243 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:15.094316 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:15.095406 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:15.095410 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:15.095413 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:33:15.095424 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:15.095445 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:33:15.095454 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:15.095557 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:15.095698 (+ 141us) server_negotiation.cc:299] Negotiation successful
0925 16:33:15.095743 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 16:33:15.096781 5490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:15.091916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41271 (local address 127.0.98.3:40453)
0925 16:33:15.092686 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:15.092751 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:15.093979 (+ 1228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:15.094481 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:15.094488 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:15.094575 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:15.095163 (+ 588us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 16:33:15.095164 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:15.095205 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 16:33:15.095271 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:33:15.095545 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:33:15.095553 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:33:15.095558 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 16:33:15.095757 (+ 199us) client_negotiation.cc:240] Negotiation successful
0925 16:33:15.096020 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":140,"threads_started":1}
I20260925 16:33:15.097891 5488 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41271
I20260925 16:33:15.098107 5488 heartbeater.cc:464] Registering TS with master...
I20260925 16:33:15.098506 5488 heartbeater.cc:511] Master 127.0.98.62:41271 requested a full tablet report, sending...
I20260925 16:33:15.099489 5038 ts_manager.cc:194] Registered new tserver with Master: 5f6fa48b06a84617ae2823a9e4ccadf0 (127.0.98.3:34127)
I20260925 16:33:15.104596 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 16:33:15.107867 5077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:15.106376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60422 (local address 127.0.98.62:41271)
0925 16:33:15.106489 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:15.106492 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:15.106673 (+ 181us) server_negotiation.cc:407] Connection header received
0925 16:33:15.106774 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:15.106776 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:15.106806 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:15.106872 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:15.107581 (+ 709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:15.107583 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:15.107584 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:33:15.107592 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:15.107605 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:33:15.107611 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:15.107671 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:15.107747 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 16:33:15.107776 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 16:33:15.110463 5038 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60422:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 16:33:15.113363 5038 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:33:15.125177 5500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:15.121789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42116 (local address 127.0.98.1:37225)
0925 16:33:15.123041 (+ 1252us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:15.123055 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:15.123079 (+ 24us) server_negotiation.cc:407] Connection header received
0925 16:33:15.123213 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:15.123217 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:15.123428 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:15.123880 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:15.124582 (+ 702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:15.124591 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:15.124605 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:33:15.124619 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:15.124671 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:33:15.124706 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:15.124827 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:15.124958 (+ 131us) server_negotiation.cc:299] Negotiation successful
0925 16:33:15.125031 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1147,"thread_start_us":121,"threads_started":1}
I20260925 16:33:15.125159 5499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:15.121618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37225 (local address 127.0.0.1:42116)
0925 16:33:15.122002 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:15.122058 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:15.122157 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:15.123562 (+ 1405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:15.123564 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:15.123580 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:15.124317 (+ 737us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 16:33:15.124320 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:15.124342 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 16:33:15.124415 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:33:15.124796 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:33:15.124806 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:33:15.124811 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 16:33:15.124928 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 16:33:15.124981 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":102,"threads_started":1}
I20260925 16:33:15.125164 5501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:15.122005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46003 (local address 127.0.0.1:33094)
0925 16:33:15.122743 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:15.122758 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:15.122829 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:15.123491 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:15.123500 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:15.123580 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:15.124447 (+ 867us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 16:33:15.124449 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:15.124459 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 16:33:15.124482 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:33:15.124819 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:33:15.124825 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:33:15.124831 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 16:33:15.124905 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 16:33:15.124979 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"thread_start_us":97,"threads_started":1}
I20260925 16:33:15.125801 5503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:15.122430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33094 (local address 127.0.98.2:46003)
0925 16:33:15.123023 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:15.123034 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:15.123075 (+ 41us) server_negotiation.cc:407] Connection header received
0925 16:33:15.123135 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:15.123141 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:15.123362 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:15.123492 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:15.124587 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:15.124591 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:15.124602 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:33:15.124612 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:15.124674 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:33:15.124708 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:15.125539 (+ 831us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:15.125585 (+ 46us) server_negotiation.cc:299] Negotiation successful
0925 16:33:15.125656 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":121,"threads_started":1}
I20260925 16:33:15.126900 5502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:15.122371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34127 (local address 127.0.0.1:50684)
0925 16:33:15.122991 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:15.123003 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:15.123080 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:15.124316 (+ 1236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:15.124319 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:15.124334 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:15.125859 (+ 1525us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 16:33:15.125861 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:15.125871 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 16:33:15.125895 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:33:15.126666 (+ 771us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:33:15.126669 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:33:15.126673 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 16:33:15.126758 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 16:33:15.126792 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":85,"threads_started":1}
I20260925 16:33:15.127023 5504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:15.122566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50684 (local address 127.0.98.3:34127)
0925 16:33:15.122940 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:15.122952 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:15.123087 (+ 135us) server_negotiation.cc:407] Connection header received
0925 16:33:15.123161 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:15.123165 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:15.123348 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:15.123492 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:15.126032 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:15.126037 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:15.126048 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:33:15.126059 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:15.126099 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:33:15.126131 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:15.126244 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:15.126800 (+ 556us) server_negotiation.cc:299] Negotiation successful
0925 16:33:15.126868 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":149,"threads_started":1}
I20260925 16:33:15.128518 5160 tablet_service.cc:1467] Processing CreateTablet for tablet 01ead14d4d4e4bc5a9970b560246cc8b (DEFAULT_TABLE table=test-table [id=3ea5c0712fa248fc9b559350d3003a69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:33:15.128518 5291 tablet_service.cc:1467] Processing CreateTablet for tablet 01ead14d4d4e4bc5a9970b560246cc8b (DEFAULT_TABLE table=test-table [id=3ea5c0712fa248fc9b559350d3003a69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:33:15.129426 5160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01ead14d4d4e4bc5a9970b560246cc8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:33:15.129506 5291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01ead14d4d4e4bc5a9970b560246cc8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:33:15.129453 5422 tablet_service.cc:1467] Processing CreateTablet for tablet 01ead14d4d4e4bc5a9970b560246cc8b (DEFAULT_TABLE table=test-table [id=3ea5c0712fa248fc9b559350d3003a69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:33:15.130335 5422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01ead14d4d4e4bc5a9970b560246cc8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:33:15.135576 5505 tablet_bootstrap.cc:492] T 01ead14d4d4e4bc5a9970b560246cc8b P afa8120b5a8e4b3d9d311028d23b90f1: Bootstrap starting.
I20260925 16:33:15.137121 5506 tablet_bootstrap.cc:492] T 01ead14d4d4e4bc5a9970b560246cc8b P 1b38e884ed684c4b97742ff7b815fb75: Bootstrap starting.
I20260925 16:33:15.137562 5507 tablet_bootstrap.cc:492] T 01ead14d4d4e4bc5a9970b560246cc8b P 5f6fa48b06a84617ae2823a9e4ccadf0: Bootstrap starting.
I20260925 16:33:15.137868 5505 tablet_bootstrap.cc:654] T 01ead14d4d4e4bc5a9970b560246cc8b P afa8120b5a8e4b3d9d311028d23b90f1: Neither blocks nor log segments found. Creating new log.
I20260925 16:33:15.138564 5505 log.cc:824] T 01ead14d4d4e4bc5a9970b560246cc8b P afa8120b5a8e4b3d9d311028d23b90f1: Log is configured to *not* fsync() on all Append() calls
I20260925 16:33:15.139452 5507 tablet_bootstrap.cc:654] T 01ead14d4d4e4bc5a9970b560246cc8b P 5f6fa48b06a84617ae2823a9e4ccadf0: Neither blocks nor log segments found. Creating new log.
I20260925 16:33:15.140247 5507 log.cc:824] T 01ead14d4d4e4bc5a9970b560246cc8b P 5f6fa48b06a84617ae2823a9e4ccadf0: Log is configured to *not* fsync() on all Append() calls
I20260925 16:33:15.140596 5506 tablet_bootstrap.cc:654] T 01ead14d4d4e4bc5a9970b560246cc8b P 1b38e884ed684c4b97742ff7b815fb75: Neither blocks nor log segments found. Creating new log.
I20260925 16:33:15.140623 5505 tablet_bootstrap.cc:492] T 01ead14d4d4e4bc5a9970b560246cc8b P afa8120b5a8e4b3d9d311028d23b90f1: No bootstrap required, opened a new log
I20260925 16:33:15.140812 5505 ts_tablet_manager.cc:1397] T 01ead14d4d4e4bc5a9970b560246cc8b P afa8120b5a8e4b3d9d311028d23b90f1: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20260925 16:33:15.141358 5506 log.cc:824] T 01ead14d4d4e4bc5a9970b560246cc8b P 1b38e884ed684c4b97742ff7b815fb75: Log is configured to *not* fsync() on all Append() calls
I20260925 16:33:15.142086 5507 tablet_bootstrap.cc:492] T 01ead14d4d4e4bc5a9970b560246cc8b P 5f6fa48b06a84617ae2823a9e4ccadf0: No bootstrap required, opened a new log
I20260925 16:33:15.142266 5507 ts_tablet_manager.cc:1397] T 01ead14d4d4e4bc5a9970b560246cc8b P 5f6fa48b06a84617ae2823a9e4ccadf0: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 16:33:15.143213 5506 tablet_bootstrap.cc:492] T 01ead14d4d4e4bc5a9970b560246cc8b P 1b38e884ed684c4b97742ff7b815fb75: No bootstrap required, opened a new log
I20260925 16:33:15.143378 5506 ts_tablet_manager.cc:1397] T 01ead14d4d4e4bc5a9970b560246cc8b P 1b38e884ed684c4b97742ff7b815fb75: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 16:33:15.144582 5505 raft_consensus.cc:348] T 01ead14d4d4e4bc5a9970b560246cc8b P afa8120b5a8e4b3d9d311028d23b90f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afa8120b5a8e4b3d9d311028d23b90f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37225 } } peers { permanent_uuid: "5f6fa48b06a84617ae2823a9e4ccadf0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34127 } } peers { permanent_uuid: "1b38e884ed684c4b97742ff7b815fb75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46003 } }
I20260925 16:33:15.144817 5505 raft_consensus.cc:374] T 01ead14d4d4e4bc5a9970b560246cc8b P afa8120b5a8e4b3d9d311028d23b90f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:33:15.144866 5505 raft_consensus.cc:729] T 01ead14d4d4e4bc5a9970b560246cc8b P afa8120b5a8e4b3d9d311028d23b90f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afa8120b5a8e4b3d9d311028d23b90f1, State: Initialized, Role: FOLLOWER
I20260925 16:33:15.145295 5505 consensus_queue.cc:260] T 01ead14d4d4e4bc5a9970b560246cc8b P afa8120b5a8e4b3d9d311028d23b90f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afa8120b5a8e4b3d9d311028d23b90f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37225 } } peers { permanent_uuid: "5f6fa48b06a84617ae2823a9e4ccadf0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34127 } } peers { permanent_uuid: "1b38e884ed684c4b97742ff7b815fb75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46003 } }
I20260925 16:33:15.146101 5226 heartbeater.cc:503] Master 127.0.98.62:41271 was elected leader, sending a full tablet report...
I20260925 16:33:15.146288 5505 ts_tablet_manager.cc:1428] T 01ead14d4d4e4bc5a9970b560246cc8b P afa8120b5a8e4b3d9d311028d23b90f1: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260925 16:33:15.146824 5507 raft_consensus.cc:348] T 01ead14d4d4e4bc5a9970b560246cc8b P 5f6fa48b06a84617ae2823a9e4ccadf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afa8120b5a8e4b3d9d311028d23b90f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37225 } } peers { permanent_uuid: "5f6fa48b06a84617ae2823a9e4ccadf0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34127 } } peers { permanent_uuid: "1b38e884ed684c4b97742ff7b815fb75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46003 } }
I20260925 16:33:15.147061 5507 raft_consensus.cc:374] T 01ead14d4d4e4bc5a9970b560246cc8b P 5f6fa48b06a84617ae2823a9e4ccadf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:33:15.147114 5507 raft_consensus.cc:729] T 01ead14d4d4e4bc5a9970b560246cc8b P 5f6fa48b06a84617ae2823a9e4ccadf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f6fa48b06a84617ae2823a9e4ccadf0, State: Initialized, Role: FOLLOWER
I20260925 16:33:15.147583 5507 consensus_queue.cc:260] T 01ead14d4d4e4bc5a9970b560246cc8b P 5f6fa48b06a84617ae2823a9e4ccadf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afa8120b5a8e4b3d9d311028d23b90f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37225 } } peers { permanent_uuid: "5f6fa48b06a84617ae2823a9e4ccadf0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34127 } } peers { permanent_uuid: "1b38e884ed684c4b97742ff7b815fb75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46003 } }
I20260925 16:33:15.147809 5506 raft_consensus.cc:348] T 01ead14d4d4e4bc5a9970b560246cc8b P 1b38e884ed684c4b97742ff7b815fb75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afa8120b5a8e4b3d9d311028d23b90f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37225 } } peers { permanent_uuid: "5f6fa48b06a84617ae2823a9e4ccadf0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34127 } } peers { permanent_uuid: "1b38e884ed684c4b97742ff7b815fb75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46003 } }
I20260925 16:33:15.148048 5506 raft_consensus.cc:374] T 01ead14d4d4e4bc5a9970b560246cc8b P 1b38e884ed684c4b97742ff7b815fb75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:33:15.148109 5506 raft_consensus.cc:729] T 01ead14d4d4e4bc5a9970b560246cc8b P 1b38e884ed684c4b97742ff7b815fb75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b38e884ed684c4b97742ff7b815fb75, State: Initialized, Role: FOLLOWER
I20260925 16:33:15.148384 5488 heartbeater.cc:503] Master 127.0.98.62:41271 was elected leader, sending a full tablet report...
I20260925 16:33:15.148535 5507 ts_tablet_manager.cc:1428] T 01ead14d4d4e4bc5a9970b560246cc8b P 5f6fa48b06a84617ae2823a9e4ccadf0: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 16:33:15.148541 5506 consensus_queue.cc:260] T 01ead14d4d4e4bc5a9970b560246cc8b P 1b38e884ed684c4b97742ff7b815fb75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afa8120b5a8e4b3d9d311028d23b90f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37225 } } peers { permanent_uuid: "5f6fa48b06a84617ae2823a9e4ccadf0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34127 } } peers { permanent_uuid: "1b38e884ed684c4b97742ff7b815fb75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46003 } }
I20260925 16:33:15.149104 5357 heartbeater.cc:503] Master 127.0.98.62:41271 was elected leader, sending a full tablet report...
I20260925 16:33:15.149431 5506 ts_tablet_manager.cc:1428] T 01ead14d4d4e4bc5a9970b560246cc8b P 1b38e884ed684c4b97742ff7b815fb75: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260925 16:33:15.166065 5358 tablet.cc:2369] T 01ead14d4d4e4bc5a9970b560246cc8b P 1b38e884ed684c4b97742ff7b815fb75: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:33:15.197724 5511 raft_consensus.cc:482] T 01ead14d4d4e4bc5a9970b560246cc8b P afa8120b5a8e4b3d9d311028d23b90f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:33:15.197990 5511 raft_consensus.cc:504] T 01ead14d4d4e4bc5a9970b560246cc8b P afa8120b5a8e4b3d9d311028d23b90f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afa8120b5a8e4b3d9d311028d23b90f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37225 } } peers { permanent_uuid: "5f6fa48b06a84617ae2823a9e4ccadf0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34127 } } peers { permanent_uuid: "1b38e884ed684c4b97742ff7b815fb75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46003 } }
I20260925 16:33:15.199265 5511 leader_election.cc:288] T 01ead14d4d4e4bc5a9970b560246cc8b P afa8120b5a8e4b3d9d311028d23b90f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f6fa48b06a84617ae2823a9e4ccadf0 (127.0.98.3:34127), 1b38e884ed684c4b97742ff7b815fb75 (127.0.98.2:46003)
I20260925 16:33:15.201325 5228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:15.199479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34127 (local address 127.0.98.1:35727)
0925 16:33:15.199656 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:15.199672 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:15.199755 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:15.200132 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:15.200134 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:15.200147 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:15.200715 (+ 568us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 16:33:15.200718 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:15.200732 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 16:33:15.200767 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:33:15.201042 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:33:15.201045 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:33:15.201049 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 16:33:15.201145 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 16:33:15.201177 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101}
I20260925 16:33:15.201354 5504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:15.199671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35727 (local address 127.0.98.3:34127)
0925 16:33:15.199826 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:15.199833 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:15.199847 (+ 14us) server_negotiation.cc:407] Connection header received
0925 16:33:15.199889 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:15.199892 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:15.199956 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:15.200034 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:15.200901 (+ 867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:15.200905 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:15.200908 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:33:15.200920 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:15.200940 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:33:15.200950 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:15.201042 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:15.201150 (+ 108us) server_negotiation.cc:299] Negotiation successful
0925 16:33:15.201194 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 16:33:15.201581 5514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:15.199522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46003 (local address 127.0.98.1:59039)
0925 16:33:15.200035 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:15.200049 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:15.200177 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:15.200481 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:15.200484 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:15.200498 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:15.200899 (+ 401us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 16:33:15.200902 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:15.200912 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 16:33:15.200936 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:33:15.201267 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:33:15.201270 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:33:15.201273 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 16:33:15.201390 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 16:33:15.201423 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":142,"threads_started":1}
I20260925 16:33:15.201574 5503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:15.199894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59039 (local address 127.0.98.2:46003)
0925 16:33:15.200051 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:15.200055 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:15.200117 (+ 62us) server_negotiation.cc:407] Connection header received
0925 16:33:15.200317 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:15.200322 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:15.200365 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:15.200468 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:15.201091 (+ 623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:15.201119 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:15.201124 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:33:15.201136 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:15.201154 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:33:15.201164 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:15.201265 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:15.201395 (+ 130us) server_negotiation.cc:299] Negotiation successful
0925 16:33:15.201439 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 16:33:15.202064 5442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ead14d4d4e4bc5a9970b560246cc8b" candidate_uuid: "afa8120b5a8e4b3d9d311028d23b90f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f6fa48b06a84617ae2823a9e4ccadf0" is_pre_election: true
I20260925 16:33:15.202174 5311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ead14d4d4e4bc5a9970b560246cc8b" candidate_uuid: "afa8120b5a8e4b3d9d311028d23b90f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b38e884ed684c4b97742ff7b815fb75" is_pre_election: true
I20260925 16:33:15.202416 5442 raft_consensus.cc:2454] T 01ead14d4d4e4bc5a9970b560246cc8b P 5f6fa48b06a84617ae2823a9e4ccadf0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate afa8120b5a8e4b3d9d311028d23b90f1 in term 0.
I20260925 16:33:15.202490 5311 raft_consensus.cc:2454] T 01ead14d4d4e4bc5a9970b560246cc8b P 1b38e884ed684c4b97742ff7b815fb75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate afa8120b5a8e4b3d9d311028d23b90f1 in term 0.
I20260925 16:33:15.202940 5115 leader_election.cc:302] T 01ead14d4d4e4bc5a9970b560246cc8b P afa8120b5a8e4b3d9d311028d23b90f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1b38e884ed684c4b97742ff7b815fb75, 5f6fa48b06a84617ae2823a9e4ccadf0, afa8120b5a8e4b3d9d311028d23b90f1; no voters:
I20260925 16:33:15.203240 5511 raft_consensus.cc:2781] T 01ead14d4d4e4bc5a9970b560246cc8b P afa8120b5a8e4b3d9d311028d23b90f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:33:15.203323 5511 raft_consensus.cc:482] T 01ead14d4d4e4bc5a9970b560246cc8b P afa8120b5a8e4b3d9d311028d23b90f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:33:15.203365 5511 raft_consensus.cc:3037] T 01ead14d4d4e4bc5a9970b560246cc8b P afa8120b5a8e4b3d9d311028d23b90f1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:33:15.204380 5511 raft_consensus.cc:504] T 01ead14d4d4e4bc5a9970b560246cc8b P afa8120b5a8e4b3d9d311028d23b90f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afa8120b5a8e4b3d9d311028d23b90f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37225 } } peers { permanent_uuid: "5f6fa48b06a84617ae2823a9e4ccadf0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34127 } } peers { permanent_uuid: "1b38e884ed684c4b97742ff7b815fb75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46003 } }
I20260925 16:33:15.204730 5511 leader_election.cc:288] T 01ead14d4d4e4bc5a9970b560246cc8b P afa8120b5a8e4b3d9d311028d23b90f1 [CANDIDATE]: Term 1 election: Requested vote from peers 5f6fa48b06a84617ae2823a9e4ccadf0 (127.0.98.3:34127), 1b38e884ed684c4b97742ff7b815fb75 (127.0.98.2:46003)
I20260925 16:33:15.205057 5442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ead14d4d4e4bc5a9970b560246cc8b" candidate_uuid: "afa8120b5a8e4b3d9d311028d23b90f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f6fa48b06a84617ae2823a9e4ccadf0"
I20260925 16:33:15.205122 5311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ead14d4d4e4bc5a9970b560246cc8b" candidate_uuid: "afa8120b5a8e4b3d9d311028d23b90f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b38e884ed684c4b97742ff7b815fb75"
I20260925 16:33:15.205202 5442 raft_consensus.cc:3037] T 01ead14d4d4e4bc5a9970b560246cc8b P 5f6fa48b06a84617ae2823a9e4ccadf0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:33:15.205264 5311 raft_consensus.cc:3037] T 01ead14d4d4e4bc5a9970b560246cc8b P 1b38e884ed684c4b97742ff7b815fb75 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:33:15.206388 5311 raft_consensus.cc:2454] T 01ead14d4d4e4bc5a9970b560246cc8b P 1b38e884ed684c4b97742ff7b815fb75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afa8120b5a8e4b3d9d311028d23b90f1 in term 1.
I20260925 16:33:15.206403 5442 raft_consensus.cc:2454] T 01ead14d4d4e4bc5a9970b560246cc8b P 5f6fa48b06a84617ae2823a9e4ccadf0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afa8120b5a8e4b3d9d311028d23b90f1 in term 1.
I20260925 16:33:15.206704 5115 leader_election.cc:302] T 01ead14d4d4e4bc5a9970b560246cc8b P afa8120b5a8e4b3d9d311028d23b90f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f6fa48b06a84617ae2823a9e4ccadf0, afa8120b5a8e4b3d9d311028d23b90f1; no voters:
I20260925 16:33:15.206890 5511 raft_consensus.cc:2781] T 01ead14d4d4e4bc5a9970b560246cc8b P afa8120b5a8e4b3d9d311028d23b90f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:33:15.207124 5511 raft_consensus.cc:686] T 01ead14d4d4e4bc5a9970b560246cc8b P afa8120b5a8e4b3d9d311028d23b90f1 [term 1 LEADER]: Becoming Leader. State: Replica: afa8120b5a8e4b3d9d311028d23b90f1, State: Running, Role: LEADER
I20260925 16:33:15.207464 5511 consensus_queue.cc:237] T 01ead14d4d4e4bc5a9970b560246cc8b P afa8120b5a8e4b3d9d311028d23b90f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afa8120b5a8e4b3d9d311028d23b90f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37225 } } peers { permanent_uuid: "5f6fa48b06a84617ae2823a9e4ccadf0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34127 } } peers { permanent_uuid: "1b38e884ed684c4b97742ff7b815fb75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46003 } }
I20260925 16:33:15.210932 5037 catalog_manager.cc:5495] T 01ead14d4d4e4bc5a9970b560246cc8b P afa8120b5a8e4b3d9d311028d23b90f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to afa8120b5a8e4b3d9d311028d23b90f1 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "afa8120b5a8e4b3d9d311028d23b90f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afa8120b5a8e4b3d9d311028d23b90f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f6fa48b06a84617ae2823a9e4ccadf0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b38e884ed684c4b97742ff7b815fb75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46003 } health_report { overall_health: UNKNOWN } } }
I20260925 16:33:15.225226 5500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:15.223856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42128 (local address 127.0.98.1:37225)
0925 16:33:15.223985 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:15.223989 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:15.224022 (+ 33us) server_negotiation.cc:407] Connection header received
0925 16:33:15.224182 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:15.224185 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:15.224246 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:15.224318 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:15.224906 (+ 588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:15.224909 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:15.224911 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:33:15.224921 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:15.224938 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:33:15.224946 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:15.225017 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:15.225089 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 16:33:15.225121 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 16:33:15.231060 5311 raft_consensus.cc:1261] T 01ead14d4d4e4bc5a9970b560246cc8b P 1b38e884ed684c4b97742ff7b815fb75 [term 1 FOLLOWER]: Refusing update from remote peer afa8120b5a8e4b3d9d311028d23b90f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:33:15.231060 5442 raft_consensus.cc:1261] T 01ead14d4d4e4bc5a9970b560246cc8b P 5f6fa48b06a84617ae2823a9e4ccadf0 [term 1 FOLLOWER]: Refusing update from remote peer afa8120b5a8e4b3d9d311028d23b90f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:33:15.231772 5511 consensus_queue.cc:1035] T 01ead14d4d4e4bc5a9970b560246cc8b P afa8120b5a8e4b3d9d311028d23b90f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f6fa48b06a84617ae2823a9e4ccadf0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:33:15.231958 5515 consensus_queue.cc:1035] T 01ead14d4d4e4bc5a9970b560246cc8b P afa8120b5a8e4b3d9d311028d23b90f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b38e884ed684c4b97742ff7b815fb75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 16:33:15.236732 5227 tablet.cc:2369] T 01ead14d4d4e4bc5a9970b560246cc8b P afa8120b5a8e4b3d9d311028d23b90f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:33:15.239770 5519 mvcc.cc:205] Tried to move back new op lower bound from 7333289964459233280 to 7333289964372594688. Current Snapshot: MvccSnapshot[applied={T|T < 7333289964459233280}]
I20260925 16:33:15.248764 5037 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:33:15.248950 5037 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:33:15.251471 5037 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 812db122f8e04cfeace7cd6fbe689252: Sending DeleteTablet for 3 replicas of tablet 01ead14d4d4e4bc5a9970b560246cc8b
I20260925 16:33:15.252305 5291 tablet_service.cc:1514] Processing DeleteTablet for tablet 01ead14d4d4e4bc5a9970b560246cc8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:33:15 UTC) from {username='slave'} at 127.0.0.1:33094
I20260925 16:33:15.252337 5160 tablet_service.cc:1514] Processing DeleteTablet for tablet 01ead14d4d4e4bc5a9970b560246cc8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:33:15 UTC) from {username='slave'} at 127.0.0.1:42116
I20260925 16:33:15.252493 5422 tablet_service.cc:1514] Processing DeleteTablet for tablet 01ead14d4d4e4bc5a9970b560246cc8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:33:15 UTC) from {username='slave'} at 127.0.0.1:50684
I20260925 16:33:15.252827 5527 tablet_replica.cc:330] T 01ead14d4d4e4bc5a9970b560246cc8b P afa8120b5a8e4b3d9d311028d23b90f1: stopping tablet replica
I20260925 16:33:15.252964 5529 tablet_replica.cc:330] T 01ead14d4d4e4bc5a9970b560246cc8b P 5f6fa48b06a84617ae2823a9e4ccadf0: stopping tablet replica
I20260925 16:33:15.253146 5527 raft_consensus.cc:2229] T 01ead14d4d4e4bc5a9970b560246cc8b P afa8120b5a8e4b3d9d311028d23b90f1 [term 1 LEADER]: Raft consensus shutting down.
I20260925 16:33:15.253237 5529 raft_consensus.cc:2229] T 01ead14d4d4e4bc5a9970b560246cc8b P 5f6fa48b06a84617ae2823a9e4ccadf0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:33:15.253516 5527 raft_consensus.cc:2258] T 01ead14d4d4e4bc5a9970b560246cc8b P afa8120b5a8e4b3d9d311028d23b90f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:33:15.253542 5529 raft_consensus.cc:2258] T 01ead14d4d4e4bc5a9970b560246cc8b P 5f6fa48b06a84617ae2823a9e4ccadf0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:33:15.253860 5528 tablet_replica.cc:330] T 01ead14d4d4e4bc5a9970b560246cc8b P 1b38e884ed684c4b97742ff7b815fb75: stopping tablet replica
I20260925 16:33:15.254117 5528 raft_consensus.cc:2229] T 01ead14d4d4e4bc5a9970b560246cc8b P 1b38e884ed684c4b97742ff7b815fb75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:33:15.254200 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 5098
I20260925 16:33:15.255831 5528 raft_consensus.cc:2258] T 01ead14d4d4e4bc5a9970b560246cc8b P 1b38e884ed684c4b97742ff7b815fb75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:33:15.256284 5529 ts_tablet_manager.cc:1905] T 01ead14d4d4e4bc5a9970b560246cc8b P 5f6fa48b06a84617ae2823a9e4ccadf0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:33:15.257498 5528 ts_tablet_manager.cc:1905] T 01ead14d4d4e4bc5a9970b560246cc8b P 1b38e884ed684c4b97742ff7b815fb75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:33:15.259776 5528 ts_tablet_manager.cc:1918] T 01ead14d4d4e4bc5a9970b560246cc8b P 1b38e884ed684c4b97742ff7b815fb75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:33:15.259770 5529 ts_tablet_manager.cc:1918] T 01ead14d4d4e4bc5a9970b560246cc8b P 5f6fa48b06a84617ae2823a9e4ccadf0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:33:15.259867 5529 log.cc:1196] T 01ead14d4d4e4bc5a9970b560246cc8b P 5f6fa48b06a84617ae2823a9e4ccadf0: Deleting WAL directory at /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353966814071-392-0/minicluster-data/ts-2/wal/wals/01ead14d4d4e4bc5a9970b560246cc8b
I20260925 16:33:15.259891 5528 log.cc:1196] T 01ead14d4d4e4bc5a9970b560246cc8b P 1b38e884ed684c4b97742ff7b815fb75: Deleting WAL directory at /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353966814071-392-0/minicluster-data/ts-1/wal/wals/01ead14d4d4e4bc5a9970b560246cc8b
I20260925 16:33:15.260200 5529 ts_tablet_manager.cc:1939] T 01ead14d4d4e4bc5a9970b560246cc8b P 5f6fa48b06a84617ae2823a9e4ccadf0: Deleting consensus metadata
I20260925 16:33:15.260224 5528 ts_tablet_manager.cc:1939] T 01ead14d4d4e4bc5a9970b560246cc8b P 1b38e884ed684c4b97742ff7b815fb75: Deleting consensus metadata
I20260925 16:33:15.261332 5023 catalog_manager.cc:4841] TS 1b38e884ed684c4b97742ff7b815fb75 (127.0.98.2:46003): tablet 01ead14d4d4e4bc5a9970b560246cc8b (table test-table [id=3ea5c0712fa248fc9b559350d3003a69]) successfully deleted
I20260925 16:33:15.261348 5024 catalog_manager.cc:4841] TS 5f6fa48b06a84617ae2823a9e4ccadf0 (127.0.98.3:34127): tablet 01ead14d4d4e4bc5a9970b560246cc8b (table test-table [id=3ea5c0712fa248fc9b559350d3003a69]) successfully deleted
W20260925 16:33:15.261791 5024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37225 (error 108)
I20260925 16:33:15.262590 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 5229
I20260925 16:33:15.262892 5502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:15.262515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37225 (local address 127.0.0.1:42136)
0925 16:33:15.262711 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:15.262792 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260925 16:33:15.264513 5024 catalog_manager.cc:4584] TS afa8120b5a8e4b3d9d311028d23b90f1 (127.0.98.1:37225): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 01ead14d4d4e4bc5a9970b560246cc8b: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37225: connect: Connection refused (error 111)
I20260925 16:33:15.268453 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 5360
I20260925 16:33:15.273748 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 5007
2026-09-25T16:33:15Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (978 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260925 16:33:15.294900 392 mini_oidc.cc:152] Starting JWKS server
I20260925 16:33:15.296470 392 webserver.cc:460] Webserver started at https://127.0.0.1:36285/ using document root <none> and password file <none>
I20260925 16:33:15.296659 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 16:33:15.296941 392 webserver.cc:460] Webserver started at http://127.0.0.1:39317/ using document root <none> and password file <none>
2026-09-25T16:33:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:33:15Z Disabled control of system clock
I20260925 16:33:15.305310 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790353966814071-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790353966814071-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790353966814071-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38485
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38485
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36285/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790353966814071-392-0/testchainca.cert with env {}
W20260925 16:33:15.412914 5536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:33:15.413187 5536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:33:15.413229 5536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:33:15.417224 5536 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:33:15.417284 5536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:33:15.417299 5536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:33:15.417315 5536 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:33:15.417330 5536 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 16:33:15.417346 5536 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 16:33:15.417361 5536 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36285/jwks.json
I20260925 16:33:15.422677 5536 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790353966814071-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790353966814071-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38485
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36285/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790353966814071-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5536
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790353966814071-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:33:15.423717 5536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:33:15.424671 5536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:33:15.433082 5541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:15.433089 5542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:15.433131 5536 server_base.cc:1034] running on GCE node
W20260925 16:33:15.433089 5544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:15.433921 5536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:33:15.435182 5536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:33:15.436367 5536 hybrid_clock.cc:648] HybridClock initialized: now 1790353995436348 us; error 32 us; skew 500 ppm
I20260925 16:33:15.438287 5536 webserver.cc:460] Webserver started at http://127.0.98.62:41735/ using document root <none> and password file <none>
I20260925 16:33:15.438936 5536 fs_manager.cc:359] Metadata directory not provided
I20260925 16:33:15.439004 5536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:33:15.439244 5536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:33:15.441346 5536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790353966814071-392-0/minicluster-data/master-0/data/instance:
uuid: "f3357c9f27dc43b19ddeeed35f99e4ba"
format_stamp: "Formatted at 2026-09-25 16:33:15 on dist-test-slave-65pf"
server_key: "acb5682a293faf8945bbb46f13b95b3e"
server_key_iv: "da02ece780914518f510c80aca566587"
server_key_version: "encryptionkey@0"
I20260925 16:33:15.441915 5536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790353966814071-392-0/minicluster-data/master-0/wal/instance:
uuid: "f3357c9f27dc43b19ddeeed35f99e4ba"
format_stamp: "Formatted at 2026-09-25 16:33:15 on dist-test-slave-65pf"
server_key: "acb5682a293faf8945bbb46f13b95b3e"
server_key_iv: "da02ece780914518f510c80aca566587"
server_key_version: "encryptionkey@0"
I20260925 16:33:15.445539 5536 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260925 16:33:15.448040 5550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:15.449260 5536 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 16:33:15.449401 5536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790353966814071-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790353966814071-392-0/minicluster-data/master-0/wal
uuid: "f3357c9f27dc43b19ddeeed35f99e4ba"
format_stamp: "Formatted at 2026-09-25 16:33:15 on dist-test-slave-65pf"
server_key: "acb5682a293faf8945bbb46f13b95b3e"
server_key_iv: "da02ece780914518f510c80aca566587"
server_key_version: "encryptionkey@0"
I20260925 16:33:15.449514 5536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790353966814071-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790353966814071-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790353966814071-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:33:15.466158 5536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:33:15.466969 5536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:33:15.467178 5536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:33:15.474224 5536 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38485
I20260925 16:33:15.474221 5602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38485 every 8 connection(s)
I20260925 16:33:15.475275 5536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:33:15.478600 5603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:33:15.480829 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 5536
I20260925 16:33:15.480930 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790353966814071-392-0/minicluster-data/master-0/wal/instance
I20260925 16:33:15.481189 392 external_mini_cluster.cc:1428] Setting key 869f4200031585a36f919e4539937114
I20260925 16:33:15.485083 5603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3357c9f27dc43b19ddeeed35f99e4ba: Bootstrap starting.
I20260925 16:33:15.487718 5603 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3357c9f27dc43b19ddeeed35f99e4ba: Neither blocks nor log segments found. Creating new log.
I20260925 16:33:15.488476 5603 log.cc:824] T 00000000000000000000000000000000 P f3357c9f27dc43b19ddeeed35f99e4ba: Log is configured to *not* fsync() on all Append() calls
I20260925 16:33:15.490500 5603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3357c9f27dc43b19ddeeed35f99e4ba: No bootstrap required, opened a new log
I20260925 16:33:15.491209 5606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:15.482658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52038 (local address 127.0.98.62:38485)
0925 16:33:15.483338 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:15.483362 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:15.483404 (+ 42us) server_negotiation.cc:407] Connection header received
0925 16:33:15.484215 (+ 811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:15.484239 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:15.484743 (+ 504us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:15.485103 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:15.485564 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:15.487017 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:15.487941 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:15.488425 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:15.488607 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:15.489410 (+ 803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:15.489415 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:15.489428 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:33:15.489825 (+ 397us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:15.489877 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:33:15.489897 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:15.490009 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:15.490133 (+ 124us) server_negotiation.cc:299] Negotiation successful
0925 16:33:15.490392 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":94,"threads_started":1}
I20260925 16:33:15.494036 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:33:15.494570 5603 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3357c9f27dc43b19ddeeed35f99e4ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3357c9f27dc43b19ddeeed35f99e4ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38485 } }
I20260925 16:33:15.494864 5603 raft_consensus.cc:374] T 00000000000000000000000000000000 P f3357c9f27dc43b19ddeeed35f99e4ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:33:15.494939 5603 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3357c9f27dc43b19ddeeed35f99e4ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3357c9f27dc43b19ddeeed35f99e4ba, State: Initialized, Role: FOLLOWER
I20260925 16:33:15.495350 5603 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3357c9f27dc43b19ddeeed35f99e4ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3357c9f27dc43b19ddeeed35f99e4ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38485 } }
I20260925 16:33:15.495501 5603 raft_consensus.cc:388] T 00000000000000000000000000000000 P f3357c9f27dc43b19ddeeed35f99e4ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:33:15.495586 5603 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3357c9f27dc43b19ddeeed35f99e4ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:33:15.495687 5603 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3357c9f27dc43b19ddeeed35f99e4ba [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:33:15.496603 5603 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3357c9f27dc43b19ddeeed35f99e4ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3357c9f27dc43b19ddeeed35f99e4ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38485 } }
I20260925 16:33:15.496948 5603 leader_election.cc:302] T 00000000000000000000000000000000 P f3357c9f27dc43b19ddeeed35f99e4ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3357c9f27dc43b19ddeeed35f99e4ba; no voters:
I20260925 16:33:15.497308 5603 leader_election.cc:288] T 00000000000000000000000000000000 P f3357c9f27dc43b19ddeeed35f99e4ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:33:15.497377 5608 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3357c9f27dc43b19ddeeed35f99e4ba [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:33:15.497637 5608 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3357c9f27dc43b19ddeeed35f99e4ba [term 1 LEADER]: Becoming Leader. State: Replica: f3357c9f27dc43b19ddeeed35f99e4ba, State: Running, Role: LEADER
I20260925 16:33:15.497973 5608 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3357c9f27dc43b19ddeeed35f99e4ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3357c9f27dc43b19ddeeed35f99e4ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38485 } }
I20260925 16:33:15.498665 5603 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3357c9f27dc43b19ddeeed35f99e4ba [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:33:15.499794 5610 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3357c9f27dc43b19ddeeed35f99e4ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3357c9f27dc43b19ddeeed35f99e4ba. Latest consensus state: current_term: 1 leader_uuid: "f3357c9f27dc43b19ddeeed35f99e4ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3357c9f27dc43b19ddeeed35f99e4ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38485 } } }
I20260925 16:33:15.499822 5609 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3357c9f27dc43b19ddeeed35f99e4ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f3357c9f27dc43b19ddeeed35f99e4ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3357c9f27dc43b19ddeeed35f99e4ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38485 } } }
I20260925 16:33:15.499914 5610 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3357c9f27dc43b19ddeeed35f99e4ba [sys.catalog]: This master's current role is: LEADER
I20260925 16:33:15.499919 5609 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3357c9f27dc43b19ddeeed35f99e4ba [sys.catalog]: This master's current role is: LEADER
I20260925 16:33:15.500258 5617 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:33:15.503389 5617 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:33:15.509022 5617 catalog_manager.cc:1347] Generated new cluster ID: 8c075713cceb472abc6a71e415b5e985
I20260925 16:33:15.509091 5617 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:33:15.520697 5617 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:33:15.521611 5617 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:33:15.527446 5617 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f3357c9f27dc43b19ddeeed35f99e4ba: Generated new TSK 0
I20260925 16:33:15.528111 5617 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 16:33:15.554066 5633 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260925 16:33:15.554266 5633 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 16:33:15.554890 5606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:15.537897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52050 (local address 127.0.98.62:38485)
0925 16:33:15.538038 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:15.538043 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:15.538307 (+ 264us) server_negotiation.cc:407] Connection header received
0925 16:33:15.538529 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:15.538532 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:15.538591 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:15.538714 (+ 123us) server_negotiation.cc:226] Negotiated authn=JWT
0925 16:33:15.538967 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:15.539475 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:15.540671 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:15.540901 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:15.540971 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:15.541320 (+ 349us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 16:33:15.554474 (+ 13154us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36285/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 16:33:15.554777 (+ 303us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52050: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36285/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":57}
W20260925 16:33:15.554991 5606 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52050: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36285/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260925 16:33:15.556526 5632 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:33:15.537829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38485 (local address 127.0.0.1:52050)
0925 16:33:15.538216 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:15.538232 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:15.538363 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:15.538738 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:15.538741 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:15.538750 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0925 16:33:15.538873 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:15.538880 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:15.539571 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:15.539575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:15.540548 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:15.540558 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:15.540988 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:15.541007 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:15.541101 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:15.541150 (+ 49us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 16:33:15.554726 (+ 13576us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36285/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 16:33:15.554982 (+ 256us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:38485: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36285/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":119,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:38485: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36285/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260925 16:33:15.557381 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 5536
2026-09-25T16:33:15Z chronyd exiting
I20260925 16:33:15.594173 392 test_util.cc:175] -----------------------------------------------
I20260925 16:33:15.594300 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790353966814071-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (300 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260925 16:33:15.595374 392 mini_oidc.cc:152] Starting JWKS server
I20260925 16:33:15.596246 392 webserver.cc:460] Webserver started at https://127.0.0.1:32957/ using document root <none> and password file <none>
I20260925 16:33:15.596380 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 16:33:15.596661 392 webserver.cc:460] Webserver started at http://127.0.0.1:41437/ using document root <none> and password file <none>
2026-09-25T16:33:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:33:15Z Disabled control of system clock
I20260925 16:33:15.605157 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790353966814071-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790353966814071-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790353966814071-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45649
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45649
--enable_jwt_token_auth=true
--jwks_url=https://localhost:32957/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790353966814071-392-0/testchainca.cert with env {}
W20260925 16:33:15.711928 5639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:33:15.712188 5639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:33:15.712280 5639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:33:15.716260 5639 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:33:15.716327 5639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:33:15.716343 5639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:33:15.716360 5639 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:33:15.716375 5639 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 16:33:15.716426 5639 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 16:33:15.716461 5639 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:32957/jwks.json
I20260925 16:33:15.721736 5639 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790353966814071-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790353966814071-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45649
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:32957/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790353966814071-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790353966814071-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:33:15.722862 5639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:33:15.723879 5639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:33:15.732232 5645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:15.732332 5639 server_base.cc:1034] running on GCE node
W20260925 16:33:15.732226 5644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:15.732226 5647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:15.733003 5639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:33:15.734086 5639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:33:15.735278 5639 hybrid_clock.cc:648] HybridClock initialized: now 1790353995735270 us; error 33 us; skew 500 ppm
I20260925 16:33:15.737159 5639 webserver.cc:460] Webserver started at http://127.0.98.62:42079/ using document root <none> and password file <none>
I20260925 16:33:15.737720 5639 fs_manager.cc:359] Metadata directory not provided
I20260925 16:33:15.737812 5639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:33:15.738029 5639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:33:15.740142 5639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790353966814071-392-0/minicluster-data/master-0/data/instance:
uuid: "77a1526222914ef6a06aa78b0466f308"
format_stamp: "Formatted at 2026-09-25 16:33:15 on dist-test-slave-65pf"
server_key: "e5f3319c95107d38e689c9f7e999214a"
server_key_iv: "471c823cb72ebb1a4a57d848ce8f8e9d"
server_key_version: "encryptionkey@0"
I20260925 16:33:15.740702 5639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790353966814071-392-0/minicluster-data/master-0/wal/instance:
uuid: "77a1526222914ef6a06aa78b0466f308"
format_stamp: "Formatted at 2026-09-25 16:33:15 on dist-test-slave-65pf"
server_key: "e5f3319c95107d38e689c9f7e999214a"
server_key_iv: "471c823cb72ebb1a4a57d848ce8f8e9d"
server_key_version: "encryptionkey@0"
I20260925 16:33:15.744189 5639 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 16:33:15.746408 5653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:15.747646 5639 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:33:15.747784 5639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790353966814071-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790353966814071-392-0/minicluster-data/master-0/wal
uuid: "77a1526222914ef6a06aa78b0466f308"
format_stamp: "Formatted at 2026-09-25 16:33:15 on dist-test-slave-65pf"
server_key: "e5f3319c95107d38e689c9f7e999214a"
server_key_iv: "471c823cb72ebb1a4a57d848ce8f8e9d"
server_key_version: "encryptionkey@0"
I20260925 16:33:15.747897 5639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790353966814071-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790353966814071-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790353966814071-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:33:15.789636 5639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:33:15.790395 5639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:33:15.790630 5639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:33:15.797435 5639 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45649
I20260925 16:33:15.797443 5705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45649 every 8 connection(s)
I20260925 16:33:15.798394 5639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:33:15.801031 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 5639
I20260925 16:33:15.801151 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790353966814071-392-0/minicluster-data/master-0/wal/instance
I20260925 16:33:15.801472 392 external_mini_cluster.cc:1428] Setting key cfd91bb6bf3a5712cca3e3ddc3b30b60
I20260925 16:33:15.801849 5706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:33:15.808523 5706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77a1526222914ef6a06aa78b0466f308: Bootstrap starting.
I20260925 16:33:15.811482 5706 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77a1526222914ef6a06aa78b0466f308: Neither blocks nor log segments found. Creating new log.
I20260925 16:33:15.812141 5709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:15.802935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45750 (local address 127.0.98.62:45649)
0925 16:33:15.803566 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:15.803581 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:15.803614 (+ 33us) server_negotiation.cc:407] Connection header received
0925 16:33:15.804636 (+ 1022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:15.804654 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:15.805040 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:15.805365 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:15.805791 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:15.806799 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:15.807924 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:15.808791 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:15.808996 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:15.810184 (+ 1188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:15.810190 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:15.810208 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:33:15.810767 (+ 559us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:15.810841 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:33:15.810874 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:15.810987 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:15.811103 (+ 116us) server_negotiation.cc:299] Negotiation successful
0925 16:33:15.811375 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":144,"threads_started":1}
I20260925 16:33:15.812227 5706 log.cc:824] T 00000000000000000000000000000000 P 77a1526222914ef6a06aa78b0466f308: Log is configured to *not* fsync() on all Append() calls
I20260925 16:33:15.814278 5706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77a1526222914ef6a06aa78b0466f308: No bootstrap required, opened a new log
I20260925 16:33:15.815068 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:33:15.818374 5706 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77a1526222914ef6a06aa78b0466f308 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77a1526222914ef6a06aa78b0466f308" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45649 } }
I20260925 16:33:15.818676 5706 raft_consensus.cc:374] T 00000000000000000000000000000000 P 77a1526222914ef6a06aa78b0466f308 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:33:15.818739 5706 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77a1526222914ef6a06aa78b0466f308 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77a1526222914ef6a06aa78b0466f308, State: Initialized, Role: FOLLOWER
I20260925 16:33:15.819139 5706 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77a1526222914ef6a06aa78b0466f308 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77a1526222914ef6a06aa78b0466f308" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45649 } }
I20260925 16:33:15.819257 5706 raft_consensus.cc:388] T 00000000000000000000000000000000 P 77a1526222914ef6a06aa78b0466f308 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:33:15.819303 5706 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77a1526222914ef6a06aa78b0466f308 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:33:15.819375 5706 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77a1526222914ef6a06aa78b0466f308 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:33:15.820192 5706 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77a1526222914ef6a06aa78b0466f308 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77a1526222914ef6a06aa78b0466f308" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45649 } }
I20260925 16:33:15.820475 5706 leader_election.cc:302] T 00000000000000000000000000000000 P 77a1526222914ef6a06aa78b0466f308 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77a1526222914ef6a06aa78b0466f308; no voters:
I20260925 16:33:15.820724 5706 leader_election.cc:288] T 00000000000000000000000000000000 P 77a1526222914ef6a06aa78b0466f308 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:33:15.820842 5711 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77a1526222914ef6a06aa78b0466f308 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:33:15.821101 5711 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77a1526222914ef6a06aa78b0466f308 [term 1 LEADER]: Becoming Leader. State: Replica: 77a1526222914ef6a06aa78b0466f308, State: Running, Role: LEADER
I20260925 16:33:15.821386 5711 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77a1526222914ef6a06aa78b0466f308 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77a1526222914ef6a06aa78b0466f308" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45649 } }
I20260925 16:33:15.821956 5706 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77a1526222914ef6a06aa78b0466f308 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:33:15.823033 5712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77a1526222914ef6a06aa78b0466f308 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "77a1526222914ef6a06aa78b0466f308" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77a1526222914ef6a06aa78b0466f308" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45649 } } }
I20260925 16:33:15.823046 5713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77a1526222914ef6a06aa78b0466f308 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77a1526222914ef6a06aa78b0466f308. Latest consensus state: current_term: 1 leader_uuid: "77a1526222914ef6a06aa78b0466f308" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77a1526222914ef6a06aa78b0466f308" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45649 } } }
I20260925 16:33:15.823171 5713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77a1526222914ef6a06aa78b0466f308 [sys.catalog]: This master's current role is: LEADER
I20260925 16:33:15.823171 5712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77a1526222914ef6a06aa78b0466f308 [sys.catalog]: This master's current role is: LEADER
I20260925 16:33:15.823580 5720 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:33:15.826783 5720 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:33:15.832576 5720 catalog_manager.cc:1347] Generated new cluster ID: 927593261db24121affa0cd90fab7efa
I20260925 16:33:15.832671 5720 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:33:15.850914 5720 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:33:15.852017 5720 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:33:15.855818 5720 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 77a1526222914ef6a06aa78b0466f308: Generated new TSK 0
I20260925 16:33:15.856581 5720 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 16:33:15.890507 5736 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260925 16:33:15.890620 5736 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 16:33:15.891368 5709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:15.874962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45752 (local address 127.0.98.62:45649)
0925 16:33:15.875111 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:15.875118 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:15.875297 (+ 179us) server_negotiation.cc:407] Connection header received
0925 16:33:15.875446 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:15.875448 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:15.875494 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:15.875585 (+ 91us) server_negotiation.cc:226] Negotiated authn=JWT
0925 16:33:15.875846 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:15.876375 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:15.877630 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:15.877839 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:15.877911 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:15.878249 (+ 338us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 16:33:15.891002 (+ 12753us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32957/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 16:33:15.891240 (+ 238us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45752: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32957/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":60}
W20260925 16:33:15.891456 5709 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45752: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32957/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260925 16:33:15.891875 5735 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:33:15.874885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45649 (local address 127.0.0.1:45752)
0925 16:33:15.875197 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:15.875213 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:15.875335 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:15.875616 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:15.875619 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:15.875627 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0925 16:33:15.875742 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:15.875748 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:15.876494 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:15.876498 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:15.877513 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:15.877524 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:15.877938 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:15.877961 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:15.878049 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:15.878076 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 16:33:15.891211 (+ 13135us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32957/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 16:33:15.891416 (+ 205us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:45649: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32957/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":104,"threads_started":1}
I20260925 16:33:15.892580 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 5639
2026-09-25T16:33:15Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (336 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260925 16:33:15.932188 392 mini_oidc.cc:152] Starting JWKS server
I20260925 16:33:15.933076 392 webserver.cc:460] Webserver started at https://127.0.0.1:42579/ using document root <none> and password file <none>
I20260925 16:33:15.933211 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 16:33:15.933475 392 webserver.cc:460] Webserver started at http://127.0.0.1:46481/ using document root <none> and password file <none>
2026-09-25T16:33:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:33:15Z Disabled control of system clock
I20260925 16:33:15.942390 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790353966814071-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790353966814071-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790353966814071-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45187
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:41743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45187
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42579/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 16:33:16.051579 5742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:33:16.051891 5742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:33:16.051932 5742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:33:16.056118 5742 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:33:16.056192 5742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:33:16.056208 5742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:33:16.056226 5742 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:33:16.056241 5742 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 16:33:16.056300 5742 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 16:33:16.056331 5742 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42579/jwks.json
I20260925 16:33:16.061208 5742 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790353966814071-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790353966814071-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45187
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42579/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5742
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790353966814071-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:33:16.062358 5742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:33:16.063858 5742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:33:16.072264 5747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:16.072335 5742 server_base.cc:1034] running on GCE node
W20260925 16:33:16.072433 5750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:16.072345 5748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:16.073045 5742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:33:16.074234 5742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:33:16.075423 5742 hybrid_clock.cc:648] HybridClock initialized: now 1790353996075413 us; error 34 us; skew 500 ppm
I20260925 16:33:16.077390 5742 webserver.cc:460] Webserver started at http://127.0.98.62:35955/ using document root <none> and password file <none>
I20260925 16:33:16.078039 5742 fs_manager.cc:359] Metadata directory not provided
I20260925 16:33:16.078132 5742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:33:16.078356 5742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:33:16.080514 5742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790353966814071-392-0/minicluster-data/master-0/data/instance:
uuid: "cf8550c4d2734b27a833f0ae250e3420"
format_stamp: "Formatted at 2026-09-25 16:33:16 on dist-test-slave-65pf"
server_key: "b8e633568da2f361fdc41c7acb987285"
server_key_iv: "83b4040652ee4cd380e5f05c098bb522"
server_key_version: "encryptionkey@0"
I20260925 16:33:16.081108 5742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790353966814071-392-0/minicluster-data/master-0/wal/instance:
uuid: "cf8550c4d2734b27a833f0ae250e3420"
format_stamp: "Formatted at 2026-09-25 16:33:16 on dist-test-slave-65pf"
server_key: "b8e633568da2f361fdc41c7acb987285"
server_key_iv: "83b4040652ee4cd380e5f05c098bb522"
server_key_version: "encryptionkey@0"
I20260925 16:33:16.084965 5742 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 16:33:16.087589 5756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:16.088858 5742 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260925 16:33:16.089000 5742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790353966814071-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790353966814071-392-0/minicluster-data/master-0/wal
uuid: "cf8550c4d2734b27a833f0ae250e3420"
format_stamp: "Formatted at 2026-09-25 16:33:16 on dist-test-slave-65pf"
server_key: "b8e633568da2f361fdc41c7acb987285"
server_key_iv: "83b4040652ee4cd380e5f05c098bb522"
server_key_version: "encryptionkey@0"
I20260925 16:33:16.089120 5742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790353966814071-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790353966814071-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790353966814071-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:33:16.119556 5742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:33:16.120327 5742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:33:16.120535 5742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:33:16.127702 5742 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45187
I20260925 16:33:16.127722 5808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45187 every 8 connection(s)
I20260925 16:33:16.128680 5742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:33:16.131896 5809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:33:16.137478 5809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf8550c4d2734b27a833f0ae250e3420: Bootstrap starting.
I20260925 16:33:16.138319 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 5742
I20260925 16:33:16.138418 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790353966814071-392-0/minicluster-data/master-0/wal/instance
I20260925 16:33:16.138767 392 external_mini_cluster.cc:1428] Setting key 92cc197ca788d94bd7ee3650e1b258af
I20260925 16:33:16.140317 5809 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf8550c4d2734b27a833f0ae250e3420: Neither blocks nor log segments found. Creating new log.
I20260925 16:33:16.141047 5809 log.cc:824] T 00000000000000000000000000000000 P cf8550c4d2734b27a833f0ae250e3420: Log is configured to *not* fsync() on all Append() calls
I20260925 16:33:16.143247 5809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf8550c4d2734b27a833f0ae250e3420: No bootstrap required, opened a new log
I20260925 16:33:16.147579 5809 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf8550c4d2734b27a833f0ae250e3420 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf8550c4d2734b27a833f0ae250e3420" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45187 } }
I20260925 16:33:16.147820 5809 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf8550c4d2734b27a833f0ae250e3420 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:33:16.147908 5809 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf8550c4d2734b27a833f0ae250e3420 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf8550c4d2734b27a833f0ae250e3420, State: Initialized, Role: FOLLOWER
I20260925 16:33:16.148087 5812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:16.139954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56934 (local address 127.0.98.62:45187)
0925 16:33:16.140639 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:16.140653 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:16.140695 (+ 42us) server_negotiation.cc:407] Connection header received
0925 16:33:16.141387 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:16.141412 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:16.141848 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:16.142150 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:16.142490 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:16.143536 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:16.144714 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:16.145166 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:16.145288 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:16.146136 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:16.146141 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:16.146154 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:33:16.146533 (+ 379us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:16.146646 (+ 113us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:33:16.146670 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:16.146752 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:16.146924 (+ 172us) server_negotiation.cc:299] Negotiation successful
0925 16:33:16.147223 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":142,"threads_started":1}
I20260925 16:33:16.148422 5809 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf8550c4d2734b27a833f0ae250e3420 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf8550c4d2734b27a833f0ae250e3420" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45187 } }
I20260925 16:33:16.148582 5809 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf8550c4d2734b27a833f0ae250e3420 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:33:16.148644 5809 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf8550c4d2734b27a833f0ae250e3420 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:33:16.148731 5809 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf8550c4d2734b27a833f0ae250e3420 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:33:16.149685 5809 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf8550c4d2734b27a833f0ae250e3420 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf8550c4d2734b27a833f0ae250e3420" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45187 } }
I20260925 16:33:16.150054 5809 leader_election.cc:302] T 00000000000000000000000000000000 P cf8550c4d2734b27a833f0ae250e3420 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf8550c4d2734b27a833f0ae250e3420; no voters:
I20260925 16:33:16.150411 5809 leader_election.cc:288] T 00000000000000000000000000000000 P cf8550c4d2734b27a833f0ae250e3420 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:33:16.150503 5814 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf8550c4d2734b27a833f0ae250e3420 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:33:16.150745 5814 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf8550c4d2734b27a833f0ae250e3420 [term 1 LEADER]: Becoming Leader. State: Replica: cf8550c4d2734b27a833f0ae250e3420, State: Running, Role: LEADER
I20260925 16:33:16.151023 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:33:16.151105 5814 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf8550c4d2734b27a833f0ae250e3420 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf8550c4d2734b27a833f0ae250e3420" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45187 } }
I20260925 16:33:16.151737 5809 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf8550c4d2734b27a833f0ae250e3420 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:33:16.152509 5815 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf8550c4d2734b27a833f0ae250e3420 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf8550c4d2734b27a833f0ae250e3420" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf8550c4d2734b27a833f0ae250e3420" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45187 } } }
I20260925 16:33:16.152624 5815 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf8550c4d2734b27a833f0ae250e3420 [sys.catalog]: This master's current role is: LEADER
I20260925 16:33:16.153026 5816 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf8550c4d2734b27a833f0ae250e3420 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf8550c4d2734b27a833f0ae250e3420. Latest consensus state: current_term: 1 leader_uuid: "cf8550c4d2734b27a833f0ae250e3420" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf8550c4d2734b27a833f0ae250e3420" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45187 } } }
I20260925 16:33:16.153132 5816 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf8550c4d2734b27a833f0ae250e3420 [sys.catalog]: This master's current role is: LEADER
I20260925 16:33:16.153388 5822 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:33:16.156394 5822 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:33:16.166488 5822 catalog_manager.cc:1347] Generated new cluster ID: c0332ebeafb64afeb1abae23799660f5
I20260925 16:33:16.166671 5822 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:33:16.190789 5822 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:33:16.191987 5822 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:33:16.201344 5822 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf8550c4d2734b27a833f0ae250e3420: Generated new TSK 0
I20260925 16:33:16.202236 5822 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:33:16.215534 5812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:16.210471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56936 (local address 127.0.98.62:45187)
0925 16:33:16.210632 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:16.210638 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:16.210848 (+ 210us) server_negotiation.cc:407] Connection header received
0925 16:33:16.211029 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:16.211031 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:16.211081 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:16.211181 (+ 100us) server_negotiation.cc:226] Negotiated authn=JWT
0925 16:33:16.211455 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:16.211991 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:16.213231 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:16.213453 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:16.213526 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:16.213870 (+ 344us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 16:33:16.215143 (+ 1273us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42579/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 16:33:16.215412 (+ 269us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56936: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42579/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":55}
W20260925 16:33:16.215623 5812 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56936: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42579/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260925 16:33:16.218390 5838 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:33:16.210397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45187 (local address 127.0.0.1:56936)
0925 16:33:16.210741 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:16.210759 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:16.210897 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:16.211201 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:16.211205 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:16.211214 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0925 16:33:16.211339 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:16.211346 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:16.212104 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:16.212109 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:16.213109 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:16.213120 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:16.213556 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:16.213581 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:16.213676 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:16.213707 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 16:33:16.215314 (+ 1607us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42579/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 16:33:16.218237 (+ 2923us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:45187: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42579/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":2666,"thread_start_us":120,"threads_started":1}
I20260925 16:33:16.219035 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 5742
E20260925 16:33:16.230106 5839 webserver.cc:533] Webserver: sslize failed:
E20260925 16:33:16.230204 5839 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-25T16:33:16Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (327 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-25T16:33:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:33:16Z Disabled control of system clock
I20260925 16:33:16.268198 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790353966814071-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790353966814071-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790353966814071-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39023
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39023
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790353966814071-392-0/testchainca.cert with env {}
W20260925 16:33:16.382730 5843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:33:16.383037 5843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:33:16.383114 5843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:33:16.387219 5843 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:33:16.387302 5843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:33:16.387318 5843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:33:16.387336 5843 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:33:16.387351 5843 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 16:33:16.387413 5843 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 16:33:16.387431 5843 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260925 16:33:16.392755 5843 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790353966814071-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790353966814071-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39023
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790353966814071-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790353966814071-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:33:16.393848 5843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:33:16.394865 5843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:33:16.403280 5848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:16.403273 5851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:16.403561 5849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:16.403595 5843 server_base.cc:1034] running on GCE node
I20260925 16:33:16.404218 5843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:33:16.405332 5843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:33:16.406546 5843 hybrid_clock.cc:648] HybridClock initialized: now 1790353996406507 us; error 25 us; skew 500 ppm
I20260925 16:33:16.408340 5843 webserver.cc:460] Webserver started at http://127.0.98.62:44019/ using document root <none> and password file <none>
I20260925 16:33:16.408913 5843 fs_manager.cc:359] Metadata directory not provided
I20260925 16:33:16.409003 5843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:33:16.409231 5843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:33:16.411345 5843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790353966814071-392-0/minicluster-data/master-0/data/instance:
uuid: "1779f238762e47d996bb7b1a2c9fbea6"
format_stamp: "Formatted at 2026-09-25 16:33:16 on dist-test-slave-65pf"
server_key: "c958d9b84c7143aa9bd7b7f2a4c3f69e"
server_key_iv: "02df1e5a9703b19e485b5229b7cd3983"
server_key_version: "encryptionkey@0"
I20260925 16:33:16.411916 5843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790353966814071-392-0/minicluster-data/master-0/wal/instance:
uuid: "1779f238762e47d996bb7b1a2c9fbea6"
format_stamp: "Formatted at 2026-09-25 16:33:16 on dist-test-slave-65pf"
server_key: "c958d9b84c7143aa9bd7b7f2a4c3f69e"
server_key_iv: "02df1e5a9703b19e485b5229b7cd3983"
server_key_version: "encryptionkey@0"
I20260925 16:33:16.415457 5843 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 16:33:16.417936 5857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:16.419206 5843 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:33:16.419341 5843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790353966814071-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790353966814071-392-0/minicluster-data/master-0/wal
uuid: "1779f238762e47d996bb7b1a2c9fbea6"
format_stamp: "Formatted at 2026-09-25 16:33:16 on dist-test-slave-65pf"
server_key: "c958d9b84c7143aa9bd7b7f2a4c3f69e"
server_key_iv: "02df1e5a9703b19e485b5229b7cd3983"
server_key_version: "encryptionkey@0"
I20260925 16:33:16.419453 5843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790353966814071-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790353966814071-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790353966814071-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:33:16.431918 5843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:33:16.432619 5843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:33:16.432817 5843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:33:16.439802 5843 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39023
I20260925 16:33:16.439843 5909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39023 every 8 connection(s)
I20260925 16:33:16.440820 5843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:33:16.443971 5910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:33:16.444212 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 5843
I20260925 16:33:16.444334 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790353966814071-392-0/minicluster-data/master-0/wal/instance
I20260925 16:33:16.444653 392 external_mini_cluster.cc:1428] Setting key e372f392665b6980b1fd9dd88ee9dcb4
I20260925 16:33:16.450552 5910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1779f238762e47d996bb7b1a2c9fbea6: Bootstrap starting.
I20260925 16:33:16.453037 5910 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1779f238762e47d996bb7b1a2c9fbea6: Neither blocks nor log segments found. Creating new log.
I20260925 16:33:16.453899 5910 log.cc:824] T 00000000000000000000000000000000 P 1779f238762e47d996bb7b1a2c9fbea6: Log is configured to *not* fsync() on all Append() calls
I20260925 16:33:16.454288 5913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:16.446012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35144 (local address 127.0.98.62:39023)
0925 16:33:16.446790 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:16.446807 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:16.446845 (+ 38us) server_negotiation.cc:407] Connection header received
0925 16:33:16.447529 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:16.447547 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:16.447941 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:16.448292 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:16.448726 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:16.449689 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:16.450638 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:16.451185 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:16.451372 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:16.452378 (+ 1006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:16.452384 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:16.452396 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:33:16.452864 (+ 468us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:16.452935 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:33:16.452957 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:16.453081 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:16.453229 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 16:33:16.453503 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":130,"threads_started":1}
I20260925 16:33:16.455891 5910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1779f238762e47d996bb7b1a2c9fbea6: No bootstrap required, opened a new log
I20260925 16:33:16.457201 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:33:16.459929 5910 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1779f238762e47d996bb7b1a2c9fbea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1779f238762e47d996bb7b1a2c9fbea6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39023 } }
I20260925 16:33:16.460207 5910 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1779f238762e47d996bb7b1a2c9fbea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:33:16.460249 5910 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1779f238762e47d996bb7b1a2c9fbea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1779f238762e47d996bb7b1a2c9fbea6, State: Initialized, Role: FOLLOWER
I20260925 16:33:16.460623 5910 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1779f238762e47d996bb7b1a2c9fbea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1779f238762e47d996bb7b1a2c9fbea6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39023 } }
I20260925 16:33:16.460742 5910 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1779f238762e47d996bb7b1a2c9fbea6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:33:16.460783 5910 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1779f238762e47d996bb7b1a2c9fbea6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:33:16.460853 5910 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1779f238762e47d996bb7b1a2c9fbea6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:33:16.461728 5910 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1779f238762e47d996bb7b1a2c9fbea6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1779f238762e47d996bb7b1a2c9fbea6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39023 } }
I20260925 16:33:16.462028 5910 leader_election.cc:302] T 00000000000000000000000000000000 P 1779f238762e47d996bb7b1a2c9fbea6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1779f238762e47d996bb7b1a2c9fbea6; no voters:
I20260925 16:33:16.462285 5910 leader_election.cc:288] T 00000000000000000000000000000000 P 1779f238762e47d996bb7b1a2c9fbea6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:33:16.462472 5915 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1779f238762e47d996bb7b1a2c9fbea6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:33:16.462770 5915 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1779f238762e47d996bb7b1a2c9fbea6 [term 1 LEADER]: Becoming Leader. State: Replica: 1779f238762e47d996bb7b1a2c9fbea6, State: Running, Role: LEADER
I20260925 16:33:16.463078 5915 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1779f238762e47d996bb7b1a2c9fbea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1779f238762e47d996bb7b1a2c9fbea6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39023 } }
I20260925 16:33:16.463673 5910 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1779f238762e47d996bb7b1a2c9fbea6 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:33:16.464756 5917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1779f238762e47d996bb7b1a2c9fbea6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1779f238762e47d996bb7b1a2c9fbea6. Latest consensus state: current_term: 1 leader_uuid: "1779f238762e47d996bb7b1a2c9fbea6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1779f238762e47d996bb7b1a2c9fbea6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39023 } } }
I20260925 16:33:16.464882 5917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1779f238762e47d996bb7b1a2c9fbea6 [sys.catalog]: This master's current role is: LEADER
I20260925 16:33:16.465265 5924 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:33:16.465663 5916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1779f238762e47d996bb7b1a2c9fbea6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1779f238762e47d996bb7b1a2c9fbea6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1779f238762e47d996bb7b1a2c9fbea6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39023 } } }
I20260925 16:33:16.465757 5916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1779f238762e47d996bb7b1a2c9fbea6 [sys.catalog]: This master's current role is: LEADER
I20260925 16:33:16.468509 5924 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:33:16.474256 5924 catalog_manager.cc:1347] Generated new cluster ID: 93ea44d1e26643ccb7cd960c50354453
I20260925 16:33:16.474329 5924 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:33:16.497334 5924 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:33:16.498687 5924 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:33:16.507035 5924 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1779f238762e47d996bb7b1a2c9fbea6: Generated new TSK 0
I20260925 16:33:16.507789 5924 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:33:16.549801 5913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:16.521075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35156 (local address 127.0.98.62:39023)
0925 16:33:16.521195 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:16.521199 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:16.521354 (+ 155us) server_negotiation.cc:407] Connection header received
0925 16:33:16.521523 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:16.521525 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:16.521574 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:16.521674 (+ 100us) server_negotiation.cc:226] Negotiated authn=JWT
0925 16:33:16.521920 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:16.522454 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:16.523722 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:16.523938 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:16.524017 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:16.524358 (+ 341us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 16:33:16.549407 (+ 25049us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 16:33:16.549678 (+ 271us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35156: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":33}
W20260925 16:33:16.549909 5913 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35156: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260925 16:33:16.550050 5939 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:33:16.520929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39023 (local address 127.0.0.1:35156)
0925 16:33:16.521257 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:16.521272 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:16.521393 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:16.521686 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:16.521689 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:16.521697 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0925 16:33:16.521817 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:16.521823 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:16.522558 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:16.522562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:16.523610 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:16.523621 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:16.524030 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:16.524054 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:16.524141 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:16.524176 (+ 35us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 16:33:16.549647 (+ 25471us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 16:33:16.549861 (+ 214us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:39023: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":114,"threads_started":1}
I20260925 16:33:16.550760 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 5843
2026-09-25T16:33:16Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (310 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (120 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (119 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:33:16 dist-test-slave-65pf krb5kdc[5951](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:33:16 dist-test-slave-65pf krb5kdc[5951](info): set up 2 sockets
Sep 25 16:33:16 dist-test-slave-65pf krb5kdc[5951](info): commencing operation
krb5kdc: starting...
W20260925 16:33:17.819739 392 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/krb5kdc/test-user.keytab.
Sep 25 16:33:17 dist-test-slave-65pf krb5kdc[5951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:33:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:33:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 16:33:17.941490 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43079
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43079
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:33:18.052476 5964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:33:18.052798 5964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:33:18.052877 5964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:33:18.056958 5964 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:33:18.057054 5964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:33:18.057094 5964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:33:18.057154 5964 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:33:18.057194 5964 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:33:18.062428 5964 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43079
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:33:18.063555 5964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:33:18.064589 5964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:33:18.073035 5970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:18.073038 5969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:18.073217 5972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:18.073509 5964 server_base.cc:1034] running on GCE node
I20260925 16:33:18.074083 5964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:33:18.075277 5964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:33:18.076470 5964 hybrid_clock.cc:648] HybridClock initialized: now 1790353998076447 us; error 36 us; skew 500 ppm
Sep 25 16:33:18 dist-test-slave-65pf krb5kdc[5951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353998, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:33:18.085101 5964 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 16:33:18.086133 5964 webserver.cc:460] Webserver started at http://127.0.98.62:32931/ using document root <none> and password file <none>
I20260925 16:33:18.086823 5964 fs_manager.cc:359] Metadata directory not provided
I20260925 16:33:18.086922 5964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:33:18.087162 5964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:33:18.089320 5964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/minicluster-data/master-0/data/instance:
uuid: "b907b9e5611344c9bddf5d5116ff98f1"
format_stamp: "Formatted at 2026-09-25 16:33:18 on dist-test-slave-65pf"
server_key: "fcc28bc6c8dd68a59f8be565cd499116"
server_key_iv: "08e7e0093db25c6b0bca36faf448da0e"
server_key_version: "encryptionkey@0"
I20260925 16:33:18.089903 5964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/minicluster-data/master-0/wal/instance:
uuid: "b907b9e5611344c9bddf5d5116ff98f1"
format_stamp: "Formatted at 2026-09-25 16:33:18 on dist-test-slave-65pf"
server_key: "fcc28bc6c8dd68a59f8be565cd499116"
server_key_iv: "08e7e0093db25c6b0bca36faf448da0e"
server_key_version: "encryptionkey@0"
I20260925 16:33:18.093710 5964 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 16:33:18.096163 5979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:18.097404 5964 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 16:33:18.097546 5964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/minicluster-data/master-0/wal
uuid: "b907b9e5611344c9bddf5d5116ff98f1"
format_stamp: "Formatted at 2026-09-25 16:33:18 on dist-test-slave-65pf"
server_key: "fcc28bc6c8dd68a59f8be565cd499116"
server_key_iv: "08e7e0093db25c6b0bca36faf448da0e"
server_key_version: "encryptionkey@0"
I20260925 16:33:18.097674 5964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:33:18.116211 5964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:33:18.119709 5964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:33:18.119937 5964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:33:18.126929 5964 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43079
I20260925 16:33:18.127234 6031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43079 every 8 connection(s)
I20260925 16:33:18.127944 5964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:33:18.131178 6032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:33:18.136675 6032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b907b9e5611344c9bddf5d5116ff98f1: Bootstrap starting.
I20260925 16:33:18.137902 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 5964
I20260925 16:33:18.138034 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/minicluster-data/master-0/wal/instance
I20260925 16:33:18.138325 392 external_mini_cluster.cc:1428] Setting key d6e8a1ece2f7428fb5a1cf4fe763bb3c
I20260925 16:33:18.139643 6032 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b907b9e5611344c9bddf5d5116ff98f1: Neither blocks nor log segments found. Creating new log.
I20260925 16:33:18.140403 6032 log.cc:824] T 00000000000000000000000000000000 P b907b9e5611344c9bddf5d5116ff98f1: Log is configured to *not* fsync() on all Append() calls
I20260925 16:33:18.142387 6032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b907b9e5611344c9bddf5d5116ff98f1: No bootstrap required, opened a new log
I20260925 16:33:18.145962 6032 raft_consensus.cc:348] T 00000000000000000000000000000000 P b907b9e5611344c9bddf5d5116ff98f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b907b9e5611344c9bddf5d5116ff98f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43079 } }
I20260925 16:33:18.146194 6032 raft_consensus.cc:374] T 00000000000000000000000000000000 P b907b9e5611344c9bddf5d5116ff98f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:33:18.146255 6032 raft_consensus.cc:729] T 00000000000000000000000000000000 P b907b9e5611344c9bddf5d5116ff98f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b907b9e5611344c9bddf5d5116ff98f1, State: Initialized, Role: FOLLOWER
I20260925 16:33:18.146783 6032 consensus_queue.cc:260] T 00000000000000000000000000000000 P b907b9e5611344c9bddf5d5116ff98f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b907b9e5611344c9bddf5d5116ff98f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43079 } }
I20260925 16:33:18.146951 6032 raft_consensus.cc:388] T 00000000000000000000000000000000 P b907b9e5611344c9bddf5d5116ff98f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:33:18.147027 6032 raft_consensus.cc:482] T 00000000000000000000000000000000 P b907b9e5611344c9bddf5d5116ff98f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:33:18.147125 6032 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b907b9e5611344c9bddf5d5116ff98f1 [term 0 FOLLOWER]: Advancing to term 1
Sep 25 16:33:18 dist-test-slave-65pf krb5kdc[5951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:33:18.148072 6032 raft_consensus.cc:504] T 00000000000000000000000000000000 P b907b9e5611344c9bddf5d5116ff98f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b907b9e5611344c9bddf5d5116ff98f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43079 } }
I20260925 16:33:18.148475 6032 leader_election.cc:302] T 00000000000000000000000000000000 P b907b9e5611344c9bddf5d5116ff98f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b907b9e5611344c9bddf5d5116ff98f1; no voters:
I20260925 16:33:18.148846 6032 leader_election.cc:288] T 00000000000000000000000000000000 P b907b9e5611344c9bddf5d5116ff98f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:33:18.148976 6037 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b907b9e5611344c9bddf5d5116ff98f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:33:18.149209 6037 raft_consensus.cc:686] T 00000000000000000000000000000000 P b907b9e5611344c9bddf5d5116ff98f1 [term 1 LEADER]: Becoming Leader. State: Replica: b907b9e5611344c9bddf5d5116ff98f1, State: Running, Role: LEADER
I20260925 16:33:18.149597 6037 consensus_queue.cc:237] T 00000000000000000000000000000000 P b907b9e5611344c9bddf5d5116ff98f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b907b9e5611344c9bddf5d5116ff98f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43079 } }
I20260925 16:33:18.150262 6032 sys_catalog.cc:564] T 00000000000000000000000000000000 P b907b9e5611344c9bddf5d5116ff98f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:33:18.151522 6039 sys_catalog.cc:455] T 00000000000000000000000000000000 P b907b9e5611344c9bddf5d5116ff98f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b907b9e5611344c9bddf5d5116ff98f1. Latest consensus state: current_term: 1 leader_uuid: "b907b9e5611344c9bddf5d5116ff98f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b907b9e5611344c9bddf5d5116ff98f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43079 } } }
I20260925 16:33:18.151522 6038 sys_catalog.cc:455] T 00000000000000000000000000000000 P b907b9e5611344c9bddf5d5116ff98f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b907b9e5611344c9bddf5d5116ff98f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b907b9e5611344c9bddf5d5116ff98f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43079 } } }
I20260925 16:33:18.151681 6038 sys_catalog.cc:458] T 00000000000000000000000000000000 P b907b9e5611344c9bddf5d5116ff98f1 [sys.catalog]: This master's current role is: LEADER
I20260925 16:33:18.151681 6039 sys_catalog.cc:458] T 00000000000000000000000000000000 P b907b9e5611344c9bddf5d5116ff98f1 [sys.catalog]: This master's current role is: LEADER
I20260925 16:33:18.152117 6046 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:33:18.153859 6035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:18.139493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34296 (local address 127.0.98.62:43079)
0925 16:33:18.139955 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:18.139969 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:18.140004 (+ 35us) server_negotiation.cc:407] Connection header received
0925 16:33:18.140709 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:18.140729 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:18.141054 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:18.141400 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:18.142232 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:18.143314 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:18.144263 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:18.144759 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:18.144928 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:18.147819 (+ 2891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:18.147832 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:18.147844 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:18.147872 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:18.149923 (+ 2051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:18.150500 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:18.150506 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:18.150512 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:18.150677 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:18.151004 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:18.151007 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:18.151010 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:18.151592 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:18.151777 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:18.152590 (+ 813us) server_negotiation.cc:299] Negotiation successful
0925 16:33:18.152893 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":129,"threads_started":1}
I20260925 16:33:18.155488 6046 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:33:18.162539 6046 catalog_manager.cc:1347] Generated new cluster ID: e9a11208466a4be7b66990fa79803fa5
I20260925 16:33:18.162655 6046 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:33:18.176967 6046 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:33:18.177976 6046 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:33:18.183820 6046 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b907b9e5611344c9bddf5d5116ff98f1: Generated new TSK 0
I20260925 16:33:18.184518 6046 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 16:33:18.214541 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43079
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:33:18.326958 6058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:33:18.327322 6058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:33:18.327412 6058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:33:18.331772 6058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:33:18.331871 6058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:33:18.331967 6058 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 16:33:18.337349 6058 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:33:18.338526 6058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:33:18.339630 6058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:33:18.349610 6063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:18.349846 6058 server_base.cc:1034] running on GCE node
W20260925 16:33:18.349607 6066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:18.349678 6064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:18.350391 6058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:33:18.351191 6058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:33:18.352368 6058 hybrid_clock.cc:648] HybridClock initialized: now 1790353998352319 us; error 70 us; skew 500 ppm
Sep 25 16:33:18 dist-test-slave-65pf krb5kdc[5951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353998, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:33:18.356161 6058 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 16:33:18.357376 6058 webserver.cc:460] Webserver started at http://127.0.98.1:44267/ using document root <none> and password file <none>
I20260925 16:33:18.358103 6058 fs_manager.cc:359] Metadata directory not provided
I20260925 16:33:18.358227 6058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:33:18.358502 6058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:33:18.361147 6058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/minicluster-data/ts-0/data/instance:
uuid: "2fe57f9a05b8406094a724ee08381d35"
format_stamp: "Formatted at 2026-09-25 16:33:18 on dist-test-slave-65pf"
server_key: "29c3ac1cbdaf6639c6c6c6f19f6db1a9"
server_key_iv: "0ab42c38bb0f53a543b27ab995ddb1c2"
server_key_version: "encryptionkey@0"
I20260925 16:33:18.361867 6058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2fe57f9a05b8406094a724ee08381d35"
format_stamp: "Formatted at 2026-09-25 16:33:18 on dist-test-slave-65pf"
server_key: "29c3ac1cbdaf6639c6c6c6f19f6db1a9"
server_key_iv: "0ab42c38bb0f53a543b27ab995ddb1c2"
server_key_version: "encryptionkey@0"
I20260925 16:33:18.366210 6058 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260925 16:33:18.368757 6073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:18.369982 6058 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:33:18.370133 6058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/minicluster-data/ts-0/wal
uuid: "2fe57f9a05b8406094a724ee08381d35"
format_stamp: "Formatted at 2026-09-25 16:33:18 on dist-test-slave-65pf"
server_key: "29c3ac1cbdaf6639c6c6c6f19f6db1a9"
server_key_iv: "0ab42c38bb0f53a543b27ab995ddb1c2"
server_key_version: "encryptionkey@0"
I20260925 16:33:18.370249 6058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:33:18.386977 6058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:33:18.389802 6058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:33:18.390017 6058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:33:18.390672 6058 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:33:18.391484 6058 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:33:18.391561 6058 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:18.391634 6058 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:33:18.391687 6058 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:18.400918 6058 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44889
I20260925 16:33:18.400939 6186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44889 every 8 connection(s)
I20260925 16:33:18.401952 6058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
Sep 25 16:33:18 dist-test-slave-65pf krb5kdc[5951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353998, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:33:18.411428 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 6058
I20260925 16:33:18.411561 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/minicluster-data/ts-0/wal/instance
I20260925 16:33:18.411859 392 external_mini_cluster.cc:1428] Setting key 03e9863697854c13ecececdbb5479b83
I20260925 16:33:18.415884 6035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:18.403913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39917 (local address 127.0.98.62:43079)
0925 16:33:18.404050 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:18.404062 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:18.404920 (+ 858us) server_negotiation.cc:407] Connection header received
0925 16:33:18.406218 (+ 1298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:18.406221 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:18.406266 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:18.406352 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:18.407638 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:18.408168 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:18.409298 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:18.409510 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:18.409583 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:18.412101 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:18.412104 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:18.412106 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:18.412137 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:18.413722 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:18.414208 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:18.414215 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:18.414220 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:18.414311 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:18.414664 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:18.414670 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:18.414675 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:18.414922 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:18.415027 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:18.415666 (+ 639us) server_negotiation.cc:299] Negotiation successful
0925 16:33:18.415767 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 16:33:18.417089 6189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:18.404303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43079 (local address 127.0.98.1:39917)
0925 16:33:18.404754 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:18.404796 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:18.406027 (+ 1231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:18.406543 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:18.406550 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:18.406987 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:18.407495 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:18.407509 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:18.408290 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:18.408294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:18.409194 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:18.409203 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:18.409636 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:18.409639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:18.409773 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:18.410313 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:18.410328 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:18.411938 (+ 1610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:33:18.413841 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:18.413846 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:18.413857 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:18.414103 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:18.414404 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:18.414406 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:18.414408 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:18.414533 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:18.415024 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:33:18.415047 (+ 23us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:33:18.415385 (+ 338us) client_negotiation.cc:769] Sending connection context
0925 16:33:18.415599 (+ 214us) client_negotiation.cc:240] Negotiation successful
0925 16:33:18.415969 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":118,"threads_started":1}
I20260925 16:33:18.418398 6187 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43079
I20260925 16:33:18.418696 6187 heartbeater.cc:464] Registering TS with master...
I20260925 16:33:18.419175 6187 heartbeater.cc:511] Master 127.0.98.62:43079 requested a full tablet report, sending...
I20260925 16:33:18.420629 5996 ts_manager.cc:194] Registered new tserver with Master: 2fe57f9a05b8406094a724ee08381d35 (127.0.98.1:44889)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 16:33:18.421778 5996 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39917
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 16:33:18.437739 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43079
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:33:18.546788 6192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:33:18.547017 6192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:33:18.547061 6192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:33:18.551095 6192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:33:18.551151 6192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:33:18.551208 6192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 16:33:18.556452 6192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:33:18.557456 6192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:33:18.558446 6192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 16:33:18.567919 6192 server_base.cc:1034] running on GCE node
W20260925 16:33:18.567909 6198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:18.567909 6197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:18.568113 6200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:18.568468 6192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:33:18.569089 6192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:33:18.570251 6192 hybrid_clock.cc:648] HybridClock initialized: now 1790353998570239 us; error 24 us; skew 500 ppm
Sep 25 16:33:18 dist-test-slave-65pf krb5kdc[5951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353998, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:33:18.573123 6192 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 16:33:18.573902 6192 webserver.cc:460] Webserver started at http://127.0.98.2:40265/ using document root <none> and password file <none>
I20260925 16:33:18.574448 6192 fs_manager.cc:359] Metadata directory not provided
I20260925 16:33:18.574508 6192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:33:18.574736 6192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:33:18.576653 6192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/minicluster-data/ts-1/data/instance:
uuid: "d33ee19caf12465e83ee23d086bc3437"
format_stamp: "Formatted at 2026-09-25 16:33:18 on dist-test-slave-65pf"
server_key: "e043a23142bed55e6d407e5a48f65c27"
server_key_iv: "86c874651720ac6bd9fba748942f27e0"
server_key_version: "encryptionkey@0"
I20260925 16:33:18.577159 6192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d33ee19caf12465e83ee23d086bc3437"
format_stamp: "Formatted at 2026-09-25 16:33:18 on dist-test-slave-65pf"
server_key: "e043a23142bed55e6d407e5a48f65c27"
server_key_iv: "86c874651720ac6bd9fba748942f27e0"
server_key_version: "encryptionkey@0"
I20260925 16:33:18.580693 6192 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 16:33:18.583101 6207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:18.584309 6192 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:33:18.584451 6192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/minicluster-data/ts-1/wal
uuid: "d33ee19caf12465e83ee23d086bc3437"
format_stamp: "Formatted at 2026-09-25 16:33:18 on dist-test-slave-65pf"
server_key: "e043a23142bed55e6d407e5a48f65c27"
server_key_iv: "86c874651720ac6bd9fba748942f27e0"
server_key_version: "encryptionkey@0"
I20260925 16:33:18.584563 6192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:33:18.595420 6192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:33:18.598719 6192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:33:18.598915 6192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:33:18.599501 6192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:33:18.600306 6192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:33:18.600378 6192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:18.600447 6192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:33:18.600502 6192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:18.609386 6192 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42649
I20260925 16:33:18.609409 6320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42649 every 8 connection(s)
I20260925 16:33:18.610338 6192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
I20260925 16:33:18.613605 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 6192
I20260925 16:33:18.613729 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/minicluster-data/ts-1/wal/instance
I20260925 16:33:18.614019 392 external_mini_cluster.cc:1428] Setting key ca69881b6894ff74476a547062dc760d
Sep 25 16:33:18 dist-test-slave-65pf krb5kdc[5951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353998, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260925 16:33:18.624902 6035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:18.612093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40711 (local address 127.0.98.62:43079)
0925 16:33:18.612267 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:18.612271 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:18.613090 (+ 819us) server_negotiation.cc:407] Connection header received
0925 16:33:18.614545 (+ 1455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:18.614563 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:18.614655 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:18.614757 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:18.616313 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:18.616832 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:18.617991 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:18.618219 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:18.618331 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:18.621130 (+ 2799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:18.621134 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:18.621143 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:18.621167 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:18.622732 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:18.623297 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:18.623301 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:18.623303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:18.623381 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:18.623738 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:18.623742 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:18.623744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:18.623990 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:18.624094 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:18.624578 (+ 484us) server_negotiation.cc:299] Negotiation successful
0925 16:33:18.624740 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260925 16:33:18.625797 6323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:18.612497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43079 (local address 127.0.98.2:40711)
0925 16:33:18.612932 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:18.612974 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:18.614327 (+ 1353us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:18.615321 (+ 994us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:18.615328 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:18.615674 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:18.616147 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:18.616161 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:18.616958 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:18.616961 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:18.617857 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:18.617866 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:18.618367 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:18.618371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:18.618539 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:18.619244 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:18.619260 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:18.620947 (+ 1687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:33:18.622848 (+ 1901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:18.622854 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:18.622870 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:18.623182 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:18.623475 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:18.623478 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:18.623480 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:18.623635 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:18.624117 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:33:18.624123 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:33:18.624359 (+ 236us) client_negotiation.cc:769] Sending connection context
0925 16:33:18.624523 (+ 164us) client_negotiation.cc:240] Negotiation successful
0925 16:33:18.624792 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":115,"threads_started":1}
I20260925 16:33:18.627121 6321 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43079
I20260925 16:33:18.627382 6321 heartbeater.cc:464] Registering TS with master...
I20260925 16:33:18.627879 6321 heartbeater.cc:511] Master 127.0.98.62:43079 requested a full tablet report, sending...
I20260925 16:33:18.628973 5996 ts_manager.cc:194] Registered new tserver with Master: d33ee19caf12465e83ee23d086bc3437 (127.0.98.2:42649)
I20260925 16:33:18.629559 5996 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40711
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 16:33:18.641364 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43079
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:33:18.752936 6326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:33:18.753208 6326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:33:18.753252 6326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:33:18.757261 6326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:33:18.757318 6326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:33:18.757417 6326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 16:33:18.762565 6326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6326
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:33:18.763648 6326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:33:18.764688 6326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:33:18.774123 6331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:18.774278 6326 server_base.cc:1034] running on GCE node
W20260925 16:33:18.774197 6332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:18.774123 6334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:18.774868 6326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:33:18.775519 6326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:33:18.776682 6326 hybrid_clock.cc:648] HybridClock initialized: now 1790353998776667 us; error 30 us; skew 500 ppm
Sep 25 16:33:18 dist-test-slave-65pf krb5kdc[5951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353998, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:33:18.779520 6326 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 16:33:18.780398 6326 webserver.cc:460] Webserver started at http://127.0.98.3:38819/ using document root <none> and password file <none>
I20260925 16:33:18.780939 6326 fs_manager.cc:359] Metadata directory not provided
I20260925 16:33:18.780999 6326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:33:18.781239 6326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:33:18.783303 6326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/minicluster-data/ts-2/data/instance:
uuid: "d39d28f0251e41d09cd1bee612ab9060"
format_stamp: "Formatted at 2026-09-25 16:33:18 on dist-test-slave-65pf"
server_key: "5fae0847ed328a593c903159b9e74259"
server_key_iv: "e67280c6b49331b3b03c7f5d002b1474"
server_key_version: "encryptionkey@0"
I20260925 16:33:18.783874 6326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d39d28f0251e41d09cd1bee612ab9060"
format_stamp: "Formatted at 2026-09-25 16:33:18 on dist-test-slave-65pf"
server_key: "5fae0847ed328a593c903159b9e74259"
server_key_iv: "e67280c6b49331b3b03c7f5d002b1474"
server_key_version: "encryptionkey@0"
I20260925 16:33:18.787447 6326 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 16:33:18.789729 6341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:18.790998 6326 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 16:33:18.791131 6326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/minicluster-data/ts-2/wal
uuid: "d39d28f0251e41d09cd1bee612ab9060"
format_stamp: "Formatted at 2026-09-25 16:33:18 on dist-test-slave-65pf"
server_key: "5fae0847ed328a593c903159b9e74259"
server_key_iv: "e67280c6b49331b3b03c7f5d002b1474"
server_key_version: "encryptionkey@0"
I20260925 16:33:18.791235 6326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:33:18.801836 6326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:33:18.804637 6326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:33:18.804824 6326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:33:18.805335 6326 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:33:18.806082 6326 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:33:18.806155 6326 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:18.806241 6326 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:33:18.806291 6326 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:18.814940 6326 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42099
I20260925 16:33:18.814966 6454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42099 every 8 connection(s)
I20260925 16:33:18.815780 6326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
I20260925 16:33:18.817284 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 6326
I20260925 16:33:18.817395 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/minicluster-data/ts-2/wal/instance
I20260925 16:33:18.817689 392 external_mini_cluster.cc:1428] Setting key 7584226dc718a07316ba1b7393cd6873
Sep 25 16:33:18 dist-test-slave-65pf krb5kdc[5951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353998, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:33:18.829818 6035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:18.817555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46917 (local address 127.0.98.62:43079)
0925 16:33:18.817722 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:18.817742 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:18.818875 (+ 1133us) server_negotiation.cc:407] Connection header received
0925 16:33:18.820793 (+ 1918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:18.820796 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:18.820883 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:18.820971 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:18.822409 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:18.823012 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:18.824093 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:18.824293 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:18.824365 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:18.826869 (+ 2504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:18.826872 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:18.826874 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:18.826897 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:18.828130 (+ 1233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:18.828566 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:18.828569 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:18.828570 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:18.828639 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:18.828918 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:18.828920 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:18.828921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:18.829090 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:18.829162 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:18.829601 (+ 439us) server_negotiation.cc:299] Negotiation successful
0925 16:33:18.829694 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 16:33:18.830796 6457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:18.818038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43079 (local address 127.0.98.3:46917)
0925 16:33:18.818734 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:18.818790 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:18.820497 (+ 1707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:18.821180 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:18.821190 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:18.821679 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:18.822284 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:18.822298 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:18.823135 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:18.823138 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:18.823989 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:18.823998 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:18.824411 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:18.824414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:18.824544 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:18.825194 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:18.825208 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:18.826726 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:33:18.828227 (+ 1501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:18.828232 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:18.828243 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:18.828484 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:18.828716 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:18.828719 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:18.828720 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:18.828844 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:18.829172 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:33:18.829177 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:33:18.829393 (+ 216us) client_negotiation.cc:769] Sending connection context
0925 16:33:18.829543 (+ 150us) client_negotiation.cc:240] Negotiation successful
0925 16:33:18.829786 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":142,"threads_started":1}
I20260925 16:33:18.831961 6455 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43079
I20260925 16:33:18.832170 6455 heartbeater.cc:464] Registering TS with master...
I20260925 16:33:18.832592 6455 heartbeater.cc:511] Master 127.0.98.62:43079 requested a full tablet report, sending...
I20260925 16:33:18.833482 5995 ts_manager.cc:194] Registered new tserver with Master: d39d28f0251e41d09cd1bee612ab9060 (127.0.98.3:42099)
I20260925 16:33:18.834007 5995 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46917
I20260925 16:33:18.843763 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/krb5kdc/test-admin.keytab.
Sep 25 16:33:18 dist-test-slave-65pf krb5kdc[5951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:33:18.859951 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260925 16:33:18.862820 6462 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:33:18.861304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35920 (local address 127.0.98.1:44889)
0925 16:33:18.861645 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:18.861659 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:18.861691 (+ 32us) server_negotiation.cc:407] Connection header received
0925 16:33:18.861741 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:18.861745 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:18.861930 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:18.862093 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:18.862605 (+ 512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35920: BlockingRecv error: recv got EOF from 127.0.0.1:35920 (error 108)
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":108,"threads_started":1}
W20260925 16:33:18.862936 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/krb5kdc/krb5cc' of type FILE
Sep 25 16:33:18 dist-test-slave-65pf krb5kdc[5951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:33:18.863946 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 16:33:18 dist-test-slave-65pf krb5kdc[5951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:33:18.864848 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 16:33:18 dist-test-slave-65pf krb5kdc[5951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 16:33:18.874298 6462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:18.865720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35926 (local address 127.0.98.1:44889)
0925 16:33:18.865855 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:18.865858 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:18.865872 (+ 14us) server_negotiation.cc:407] Connection header received
0925 16:33:18.865983 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:18.865985 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:18.866023 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:18.866092 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:18.866570 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:18.867396 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:18.868229 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:18.868644 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:18.868730 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:18.870974 (+ 2244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:18.870981 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:18.870994 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:18.871016 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:18.872648 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:18.873077 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:18.873082 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:18.873083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:18.873166 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:18.873394 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:18.873396 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:18.873397 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:18.873619 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:18.873833 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:18.873946 (+ 113us) server_negotiation.cc:299] Negotiation successful
0925 16:33:18.874097 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260925 16:33:18.877955 6462 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:33:18.877123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35942 (local address 127.0.98.1:44889)
0925 16:33:18.877250 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:18.877254 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:18.877270 (+ 16us) server_negotiation.cc:407] Connection header received
0925 16:33:18.877377 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:18.877379 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:18.877412 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:18.877476 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:18.877859 (+ 383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35942: BlockingRecv error: recv got EOF from 127.0.0.1:35942 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260925 16:33:18.878084 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/krb5kdc/krb5cc' of type FILE
Sep 25 16:33:18 dist-test-slave-65pf krb5kdc[5951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:33:18.879123 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 16:33:18 dist-test-slave-65pf krb5kdc[5951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:33:18.879984 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 16:33:18 dist-test-slave-65pf krb5kdc[5951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 16:33:18.888846 6462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:18.880746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35956 (local address 127.0.98.1:44889)
0925 16:33:18.880862 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:18.880866 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:18.881010 (+ 144us) server_negotiation.cc:407] Connection header received
0925 16:33:18.881171 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:18.881175 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:18.881213 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:18.881306 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:18.881788 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:18.882324 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:18.883234 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:18.883475 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:18.883585 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:18.885760 (+ 2175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:18.885764 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:18.885767 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:18.885795 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:18.887364 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:18.887801 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:18.887804 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:18.887805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:18.887884 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:18.888087 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:18.888089 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:18.888091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:18.888241 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:18.888339 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:18.888604 (+ 265us) server_negotiation.cc:299] Negotiation successful
0925 16:33:18.888715 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260925 16:33:18.891398 6462 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:33:18.890501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35968 (local address 127.0.98.1:44889)
0925 16:33:18.890667 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:18.890671 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:18.890708 (+ 37us) server_negotiation.cc:407] Connection header received
0925 16:33:18.890815 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:18.890817 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:18.890849 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:18.890913 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:18.891274 (+ 361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35968: BlockingRecv error: recv got EOF from 127.0.0.1:35968 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260925 16:33:18.891463 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353966814071-392-0/krb5kdc/krb5cc' of type FILE
Sep 25 16:33:18 dist-test-slave-65pf krb5kdc[5951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:33:18.892462 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 16:33:18 dist-test-slave-65pf krb5kdc[5951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:33:18.893343 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 16:33:18 dist-test-slave-65pf krb5kdc[5951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 16:33:18.901994 6462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:18.894121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35974 (local address 127.0.98.1:44889)
0925 16:33:18.894250 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:18.894258 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:18.894271 (+ 13us) server_negotiation.cc:407] Connection header received
0925 16:33:18.894371 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:18.894373 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:18.894406 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:18.894471 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:18.895010 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:18.895484 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:18.896273 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:18.896591 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:18.896699 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:18.899050 (+ 2351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:18.899053 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:18.899056 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:18.899076 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:18.900498 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:18.900908 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:18.900910 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:18.900912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:18.900983 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:18.901208 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:18.901210 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:18.901211 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:18.901363 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:18.901456 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:18.901667 (+ 211us) server_negotiation.cc:299] Negotiation successful
0925 16:33:18.901798 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 16:33:18.903215 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 6058
I20260925 16:33:18.908651 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 6192
I20260925 16:33:18.914897 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 6326
I20260925 16:33:18.920127 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 5964
2026-09-25T16:33:18Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2131 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:33:18 dist-test-slave-65pf krb5kdc[6475](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:33:18 dist-test-slave-65pf krb5kdc[6475](info): set up 2 sockets
Sep 25 16:33:18 dist-test-slave-65pf krb5kdc[6475](info): commencing operation
krb5kdc: starting...
W20260925 16:33:19.958966 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/krb5kdc/test-user.keytab.
Sep 25 16:33:20 dist-test-slave-65pf krb5kdc[6475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:33:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:33:20Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260925 16:33:20.077399 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34575
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34575
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:33:20.184206 6488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:33:20.184509 6488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:33:20.184585 6488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:33:20.188580 6488 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:33:20.188673 6488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:33:20.188710 6488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:33:20.188746 6488 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:33:20.188777 6488 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:33:20.193845 6488 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34575
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6488
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:33:20.194901 6488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:33:20.195906 6488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:33:20.204107 6494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:20.204210 6493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:20.204138 6496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:20.204360 6488 server_base.cc:1034] running on GCE node
I20260925 16:33:20.204921 6488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:33:20.205963 6488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:33:20.207122 6488 hybrid_clock.cc:648] HybridClock initialized: now 1790354000207112 us; error 36 us; skew 500 ppm
Sep 25 16:33:20 dist-test-slave-65pf krb5kdc[6475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354000, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:33:20.209750 6488 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260925 16:33:20.210682 6488 webserver.cc:460] Webserver started at http://127.0.98.62:46603/ using document root <none> and password file <none>
I20260925 16:33:20.211196 6488 fs_manager.cc:359] Metadata directory not provided
I20260925 16:33:20.211253 6488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:33:20.211411 6488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:33:20.213366 6488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/master-0/data/instance:
uuid: "6772881645114340bec15e5a7e09e6e5"
format_stamp: "Formatted at 2026-09-25 16:33:20 on dist-test-slave-65pf"
server_key: "63082fdfc9faf4705d698ab8d2905149"
server_key_iv: "dac8ed99b728fd6be7a10e8d18d6f269"
server_key_version: "encryptionkey@0"
I20260925 16:33:20.213856 6488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/master-0/wal/instance:
uuid: "6772881645114340bec15e5a7e09e6e5"
format_stamp: "Formatted at 2026-09-25 16:33:20 on dist-test-slave-65pf"
server_key: "63082fdfc9faf4705d698ab8d2905149"
server_key_iv: "dac8ed99b728fd6be7a10e8d18d6f269"
server_key_version: "encryptionkey@0"
I20260925 16:33:20.217402 6488 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260925 16:33:20.219712 6503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:20.220916 6488 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 16:33:20.221040 6488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/master-0/wal
uuid: "6772881645114340bec15e5a7e09e6e5"
format_stamp: "Formatted at 2026-09-25 16:33:20 on dist-test-slave-65pf"
server_key: "63082fdfc9faf4705d698ab8d2905149"
server_key_iv: "dac8ed99b728fd6be7a10e8d18d6f269"
server_key_version: "encryptionkey@0"
I20260925 16:33:20.221153 6488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:33:20.234396 6488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:33:20.237085 6488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:33:20.237298 6488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:33:20.244270 6488 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34575
I20260925 16:33:20.244581 6555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34575 every 8 connection(s)
I20260925 16:33:20.245236 6488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:33:20.248172 6556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:33:20.253163 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 6488
I20260925 16:33:20.253264 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/master-0/wal/instance
I20260925 16:33:20.253436 6556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6772881645114340bec15e5a7e09e6e5: Bootstrap starting.
I20260925 16:33:20.253530 392 external_mini_cluster.cc:1428] Setting key 492205f5e3d0de5a7743a092f8ba7b63
I20260925 16:33:20.255925 6556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6772881645114340bec15e5a7e09e6e5: Neither blocks nor log segments found. Creating new log.
I20260925 16:33:20.256701 6556 log.cc:824] T 00000000000000000000000000000000 P 6772881645114340bec15e5a7e09e6e5: Log is configured to *not* fsync() on all Append() calls
I20260925 16:33:20.258519 6556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6772881645114340bec15e5a7e09e6e5: No bootstrap required, opened a new log
Sep 25 16:33:20 dist-test-slave-65pf krb5kdc[6475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 16:33:20.261962 6556 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6772881645114340bec15e5a7e09e6e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6772881645114340bec15e5a7e09e6e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34575 } }
I20260925 16:33:20.262246 6556 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6772881645114340bec15e5a7e09e6e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:33:20.262315 6556 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6772881645114340bec15e5a7e09e6e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6772881645114340bec15e5a7e09e6e5, State: Initialized, Role: FOLLOWER
I20260925 16:33:20.262801 6556 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6772881645114340bec15e5a7e09e6e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6772881645114340bec15e5a7e09e6e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34575 } }
I20260925 16:33:20.262929 6556 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6772881645114340bec15e5a7e09e6e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:33:20.263038 6556 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6772881645114340bec15e5a7e09e6e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:33:20.263172 6556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6772881645114340bec15e5a7e09e6e5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:33:20.264007 6556 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6772881645114340bec15e5a7e09e6e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6772881645114340bec15e5a7e09e6e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34575 } }
I20260925 16:33:20.264336 6556 leader_election.cc:302] T 00000000000000000000000000000000 P 6772881645114340bec15e5a7e09e6e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6772881645114340bec15e5a7e09e6e5; no voters:
I20260925 16:33:20.264634 6556 leader_election.cc:288] T 00000000000000000000000000000000 P 6772881645114340bec15e5a7e09e6e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:33:20.264729 6561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6772881645114340bec15e5a7e09e6e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:33:20.264937 6561 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6772881645114340bec15e5a7e09e6e5 [term 1 LEADER]: Becoming Leader. State: Replica: 6772881645114340bec15e5a7e09e6e5, State: Running, Role: LEADER
I20260925 16:33:20.265266 6561 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6772881645114340bec15e5a7e09e6e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6772881645114340bec15e5a7e09e6e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34575 } }
I20260925 16:33:20.265923 6556 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6772881645114340bec15e5a7e09e6e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:33:20.266904 6562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6772881645114340bec15e5a7e09e6e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6772881645114340bec15e5a7e09e6e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6772881645114340bec15e5a7e09e6e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34575 } } }
I20260925 16:33:20.267030 6562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6772881645114340bec15e5a7e09e6e5 [sys.catalog]: This master's current role is: LEADER
I20260925 16:33:20.267438 6563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6772881645114340bec15e5a7e09e6e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6772881645114340bec15e5a7e09e6e5. Latest consensus state: current_term: 1 leader_uuid: "6772881645114340bec15e5a7e09e6e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6772881645114340bec15e5a7e09e6e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34575 } } }
I20260925 16:33:20.267592 6563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6772881645114340bec15e5a7e09e6e5 [sys.catalog]: This master's current role is: LEADER
I20260925 16:33:20.267891 6570 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:33:20.268139 6559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:20.254894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34684 (local address 127.0.98.62:34575)
0925 16:33:20.255298 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:20.255313 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:20.255348 (+ 35us) server_negotiation.cc:407] Connection header received
0925 16:33:20.255941 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:20.255956 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:20.256334 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:20.256608 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:20.257500 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:20.258419 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:20.259338 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:20.259801 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:20.259979 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:20.262675 (+ 2696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:20.262681 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:20.262698 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:20.262733 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:20.264658 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:20.265196 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:20.265202 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:20.265207 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:20.265334 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:20.265642 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:20.265646 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:20.265648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:20.266139 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:20.266342 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:20.266696 (+ 354us) server_negotiation.cc:299] Negotiation successful
0925 16:33:20.267039 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":145,"threads_started":1}
I20260925 16:33:20.271037 6570 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:33:20.276172 6570 catalog_manager.cc:1347] Generated new cluster ID: 05ab274724e944f197784d0f601636d0
I20260925 16:33:20.276234 6570 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:33:20.288105 6570 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:33:20.289422 6570 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:33:20.304056 6570 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6772881645114340bec15e5a7e09e6e5: Generated new TSK 0
I20260925 16:33:20.304941 6570 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260925 16:33:20.338546 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34575
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:33:20.452620 6582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:33:20.452919 6582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:33:20.452989 6582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:33:20.457193 6582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:33:20.457283 6582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:33:20.457374 6582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 16:33:20.462509 6582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:33:20.463548 6582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:33:20.464588 6582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:33:20.473949 6590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:20.473961 6587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:20.473940 6588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:20.474346 6582 server_base.cc:1034] running on GCE node
I20260925 16:33:20.474808 6582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:33:20.475481 6582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:33:20.476665 6582 hybrid_clock.cc:648] HybridClock initialized: now 1790354000476642 us; error 44 us; skew 500 ppm
Sep 25 16:33:20 dist-test-slave-65pf krb5kdc[6475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354000, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:33:20.479336 6582 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260925 16:33:20.480129 6582 webserver.cc:460] Webserver started at http://127.0.98.1:33421/ using document root <none> and password file <none>
I20260925 16:33:20.480644 6582 fs_manager.cc:359] Metadata directory not provided
I20260925 16:33:20.480702 6582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:33:20.480863 6582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:33:20.482931 6582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-0/data/instance:
uuid: "37e3504cf5f848cc912031815590b8a8"
format_stamp: "Formatted at 2026-09-25 16:33:20 on dist-test-slave-65pf"
server_key: "1ba7b1809b1f25b9be2f46962f76475f"
server_key_iv: "27c52fc80434bc5eb335ec37be52eddf"
server_key_version: "encryptionkey@0"
I20260925 16:33:20.483443 6582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-0/wal/instance:
uuid: "37e3504cf5f848cc912031815590b8a8"
format_stamp: "Formatted at 2026-09-25 16:33:20 on dist-test-slave-65pf"
server_key: "1ba7b1809b1f25b9be2f46962f76475f"
server_key_iv: "27c52fc80434bc5eb335ec37be52eddf"
server_key_version: "encryptionkey@0"
I20260925 16:33:20.487210 6582 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 16:33:20.489761 6597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:20.491031 6582 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:33:20.491128 6582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-0/wal
uuid: "37e3504cf5f848cc912031815590b8a8"
format_stamp: "Formatted at 2026-09-25 16:33:20 on dist-test-slave-65pf"
server_key: "1ba7b1809b1f25b9be2f46962f76475f"
server_key_iv: "27c52fc80434bc5eb335ec37be52eddf"
server_key_version: "encryptionkey@0"
I20260925 16:33:20.491214 6582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:33:20.508944 6582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:33:20.512123 6582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:33:20.512288 6582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:33:20.512851 6582 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:33:20.513636 6582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:33:20.513684 6582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:20.513724 6582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:33:20.513741 6582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:20.522512 6582 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43207
I20260925 16:33:20.522531 6710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43207 every 8 connection(s)
I20260925 16:33:20.523435 6582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
I20260925 16:33:20.524798 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 6582
I20260925 16:33:20.524950 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-0/wal/instance
I20260925 16:33:20.525339 392 external_mini_cluster.cc:1428] Setting key 318d9baab1350f9394056cbc055c6d75
Sep 25 16:33:20 dist-test-slave-65pf krb5kdc[6475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354000, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
I20260925 16:33:20.539423 6559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:20.525433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35803 (local address 127.0.98.62:34575)
0925 16:33:20.525566 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:20.525574 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:20.526740 (+ 1166us) server_negotiation.cc:407] Connection header received
0925 16:33:20.528246 (+ 1506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:20.528252 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:20.528318 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:20.528417 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:20.530114 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:20.530691 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:20.532553 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:20.532785 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:20.532865 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:20.535838 (+ 2973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:20.535843 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:20.535846 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:20.535880 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:20.537244 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:20.537818 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:20.537822 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:20.537827 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:20.537930 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:20.538252 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:20.538254 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:20.538256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:20.538461 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:20.538638 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:20.539188 (+ 550us) server_negotiation.cc:299] Negotiation successful
0925 16:33:20.539297 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 16:33:20.540643 6713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:20.525935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34575 (local address 127.0.98.1:35803)
0925 16:33:20.526533 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:20.526615 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:20.527982 (+ 1367us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:20.528634 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:20.528647 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:20.529198 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:20.529947 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:20.529979 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:20.530823 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:20.530827 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:20.532419 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:20.532434 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:20.532949 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:20.532957 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:20.533198 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:20.533867 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:20.533894 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:20.535633 (+ 1739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:33:20.537356 (+ 1723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:20.537361 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:20.537374 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:20.537633 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:20.538034 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:20.538036 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:20.538038 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:20.538170 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:20.538650 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:33:20.538655 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:33:20.538929 (+ 274us) client_negotiation.cc:769] Sending connection context
0925 16:33:20.539141 (+ 212us) client_negotiation.cc:240] Negotiation successful
0925 16:33:20.539520 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":132,"threads_started":1}
I20260925 16:33:20.541949 6711 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34575
I20260925 16:33:20.542212 6711 heartbeater.cc:464] Registering TS with master...
I20260925 16:33:20.542843 6711 heartbeater.cc:511] Master 127.0.98.62:34575 requested a full tablet report, sending...
I20260925 16:33:20.544386 6520 ts_manager.cc:194] Registered new tserver with Master: 37e3504cf5f848cc912031815590b8a8 (127.0.98.1:43207)
I20260925 16:33:20.545495 6520 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35803
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260925 16:33:20.553798 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34575
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:33:20.683279 6716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:33:20.683694 6716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:33:20.683801 6716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:33:20.689569 6716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:33:20.689657 6716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:33:20.689797 6716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 16:33:20.695479 6716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:33:20.696557 6716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:33:20.697610 6716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:33:20.707322 6722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:20.707301 6724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:20.707301 6721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:20.707427 6716 server_base.cc:1034] running on GCE node
I20260925 16:33:20.707981 6716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:33:20.708590 6716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:33:20.709801 6716 hybrid_clock.cc:648] HybridClock initialized: now 1790354000709786 us; error 29 us; skew 500 ppm
Sep 25 16:33:20 dist-test-slave-65pf krb5kdc[6475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354000, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:33:20.712679 6716 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260925 16:33:20.713611 6716 webserver.cc:460] Webserver started at http://127.0.98.2:36409/ using document root <none> and password file <none>
I20260925 16:33:20.714180 6716 fs_manager.cc:359] Metadata directory not provided
I20260925 16:33:20.714274 6716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:33:20.714506 6716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:33:20.716559 6716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-1/data/instance:
uuid: "0e11f3b8048a40d093ebd83dfdfe0b46"
format_stamp: "Formatted at 2026-09-25 16:33:20 on dist-test-slave-65pf"
server_key: "c48ed1e054ed5e68dd3571ebda78efa0"
server_key_iv: "3404f5623c02d64ac6dd96e16bc7f187"
server_key_version: "encryptionkey@0"
I20260925 16:33:20.717147 6716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0e11f3b8048a40d093ebd83dfdfe0b46"
format_stamp: "Formatted at 2026-09-25 16:33:20 on dist-test-slave-65pf"
server_key: "c48ed1e054ed5e68dd3571ebda78efa0"
server_key_iv: "3404f5623c02d64ac6dd96e16bc7f187"
server_key_version: "encryptionkey@0"
I20260925 16:33:20.720961 6716 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260925 16:33:20.723377 6731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:20.724588 6716 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:33:20.724723 6716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-1/wal
uuid: "0e11f3b8048a40d093ebd83dfdfe0b46"
format_stamp: "Formatted at 2026-09-25 16:33:20 on dist-test-slave-65pf"
server_key: "c48ed1e054ed5e68dd3571ebda78efa0"
server_key_iv: "3404f5623c02d64ac6dd96e16bc7f187"
server_key_version: "encryptionkey@0"
I20260925 16:33:20.724834 6716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:33:20.735370 6716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:33:20.738886 6716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:33:20.739156 6716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:33:20.739817 6716 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:33:20.740712 6716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:33:20.740790 6716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:20.740863 6716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:33:20.740923 6716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:20.750495 6716 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43771
I20260925 16:33:20.750514 6844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43771 every 8 connection(s)
I20260925 16:33:20.751511 6716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
I20260925 16:33:20.760385 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 6716
I20260925 16:33:20.760531 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-1/wal/instance
I20260925 16:33:20.760847 392 external_mini_cluster.cc:1428] Setting key eea4fbca7ec77442f71f5bc1f052c58a
Sep 25 16:33:20 dist-test-slave-65pf krb5kdc[6475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354000, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 16:33:20.765652 6559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:20.753398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39771 (local address 127.0.98.62:34575)
0925 16:33:20.753549 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:20.753555 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:20.754460 (+ 905us) server_negotiation.cc:407] Connection header received
0925 16:33:20.755808 (+ 1348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:20.755811 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:20.755861 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:20.755949 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:20.757136 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:20.757706 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:20.758849 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:20.759077 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:20.759165 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:20.761750 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:20.761753 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:20.761755 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:20.761779 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:20.763287 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:20.763931 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:20.763937 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:20.763941 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:20.764045 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:20.764385 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:20.764392 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:20.764396 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:20.764656 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:20.764765 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:20.765244 (+ 479us) server_negotiation.cc:299] Negotiation successful
0925 16:33:20.765390 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 16:33:20.766410 6847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:20.753842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34575 (local address 127.0.98.2:39771)
0925 16:33:20.754305 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:20.754348 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:20.755598 (+ 1250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:20.756117 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:20.756124 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:20.756475 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:20.756960 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:20.756972 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:20.757822 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:20.757824 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:20.758742 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:20.758750 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:20.759217 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:20.759220 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:20.759359 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:20.759906 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:20.759922 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:20.761574 (+ 1652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:33:20.763407 (+ 1833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:20.763416 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:20.763436 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:20.763820 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:20.764147 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:20.764149 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:20.764151 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:20.764292 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:20.764760 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:33:20.764765 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:33:20.765007 (+ 242us) client_negotiation.cc:769] Sending connection context
0925 16:33:20.765164 (+ 157us) client_negotiation.cc:240] Negotiation successful
0925 16:33:20.765419 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":122,"threads_started":1}
I20260925 16:33:20.767887 6845 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34575
I20260925 16:33:20.768144 6845 heartbeater.cc:464] Registering TS with master...
I20260925 16:33:20.768635 6845 heartbeater.cc:511] Master 127.0.98.62:34575 requested a full tablet report, sending...
I20260925 16:33:20.769750 6520 ts_manager.cc:194] Registered new tserver with Master: 0e11f3b8048a40d093ebd83dfdfe0b46 (127.0.98.2:43771)
I20260925 16:33:20.770550 6520 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39771
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260925 16:33:20.789428 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34575
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:33:20.901197 6850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:33:20.901513 6850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:33:20.901557 6850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:33:20.905660 6850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:33:20.905722 6850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:33:20.905821 6850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 16:33:20.911120 6850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:33:20.912200 6850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:33:20.913241 6850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:33:20.922653 6855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:20.922797 6850 server_base.cc:1034] running on GCE node
W20260925 16:33:20.922685 6858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:20.922653 6856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:20.923429 6850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:33:20.924105 6850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:33:20.925266 6850 hybrid_clock.cc:648] HybridClock initialized: now 1790354000925234 us; error 45 us; skew 500 ppm
Sep 25 16:33:20 dist-test-slave-65pf krb5kdc[6475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354000, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:33:20.928148 6850 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260925 16:33:20.929064 6850 webserver.cc:460] Webserver started at http://127.0.98.3:43395/ using document root <none> and password file <none>
I20260925 16:33:20.929574 6850 fs_manager.cc:359] Metadata directory not provided
I20260925 16:33:20.929661 6850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:33:20.929859 6850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:33:20.931951 6850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-2/data/instance:
uuid: "8d93ff6ea52e43f295aa3c9846e05303"
format_stamp: "Formatted at 2026-09-25 16:33:20 on dist-test-slave-65pf"
server_key: "4efd14369c81f051945e0e662c4363d2"
server_key_iv: "b2b69ed619d2cd6b973de6ad7b904f1e"
server_key_version: "encryptionkey@0"
I20260925 16:33:20.932471 6850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8d93ff6ea52e43f295aa3c9846e05303"
format_stamp: "Formatted at 2026-09-25 16:33:20 on dist-test-slave-65pf"
server_key: "4efd14369c81f051945e0e662c4363d2"
server_key_iv: "b2b69ed619d2cd6b973de6ad7b904f1e"
server_key_version: "encryptionkey@0"
I20260925 16:33:20.936020 6850 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 16:33:20.938354 6865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:20.939668 6850 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:33:20.939829 6850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-2/wal
uuid: "8d93ff6ea52e43f295aa3c9846e05303"
format_stamp: "Formatted at 2026-09-25 16:33:20 on dist-test-slave-65pf"
server_key: "4efd14369c81f051945e0e662c4363d2"
server_key_iv: "b2b69ed619d2cd6b973de6ad7b904f1e"
server_key_version: "encryptionkey@0"
I20260925 16:33:20.939949 6850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:33:20.962888 6850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:33:20.966667 6850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:33:20.966956 6850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:33:20.967625 6850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:33:20.968603 6850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:33:20.968683 6850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:20.968765 6850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:33:20.968814 6850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:20.978003 6850 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42375
I20260925 16:33:20.978067 6978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42375 every 8 connection(s)
I20260925 16:33:20.979048 6850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
I20260925 16:33:20.985885 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 6850
I20260925 16:33:20.985986 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-2/wal/instance
I20260925 16:33:20.986323 392 external_mini_cluster.cc:1428] Setting key 64d73e1cb6abda7bbe74244c066949f8
Sep 25 16:33:20 dist-test-slave-65pf krb5kdc[6475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354000, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 16:33:20.992815 6559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:20.980907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52683 (local address 127.0.98.62:34575)
0925 16:33:20.981086 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:20.981090 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:20.981894 (+ 804us) server_negotiation.cc:407] Connection header received
0925 16:33:20.983362 (+ 1468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:20.983366 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:20.983412 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:20.983484 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:20.984594 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:20.985096 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:20.986269 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:20.986595 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:20.986702 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:20.989332 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:20.989339 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:20.989343 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:20.989376 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:20.991096 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:20.991567 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:20.991569 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:20.991570 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:20.991640 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:20.991922 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:20.991924 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:20.991925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:20.992090 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:20.992163 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:20.992605 (+ 442us) server_negotiation.cc:299] Negotiation successful
0925 16:33:20.992698 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 16:33:20.993667 6981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:20.981288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34575 (local address 127.0.98.3:52683)
0925 16:33:20.981730 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:20.981773 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:20.983171 (+ 1398us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:20.983637 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:20.983644 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:20.984002 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:20.984465 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:20.984477 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:20.985195 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:20.985197 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:20.986149 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:20.986157 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:20.986749 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:20.986752 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:20.986925 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:20.987638 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:20.987657 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:20.989175 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:33:20.991197 (+ 2022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:20.991202 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:20.991213 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:20.991473 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:20.991719 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:20.991721 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:20.991722 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:20.991847 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:20.992175 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:33:20.992179 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:33:20.992399 (+ 220us) client_negotiation.cc:769] Sending connection context
0925 16:33:20.992551 (+ 152us) client_negotiation.cc:240] Negotiation successful
0925 16:33:20.992794 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":123,"threads_started":1}
I20260925 16:33:20.994863 6979 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34575
I20260925 16:33:20.995088 6979 heartbeater.cc:464] Registering TS with master...
I20260925 16:33:20.995571 6979 heartbeater.cc:511] Master 127.0.98.62:34575 requested a full tablet report, sending...
I20260925 16:33:20.996611 6520 ts_manager.cc:194] Registered new tserver with Master: 8d93ff6ea52e43f295aa3c9846e05303 (127.0.98.3:42375)
I20260925 16:33:20.997141 6520 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52683
I20260925 16:33:21.000085 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260925 16:33:21.025111 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:45951
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34575,127.0.98.61:45951
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:33:21.134733 6984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:33:21.134985 6984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:33:21.135025 6984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:33:21.138972 6984 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:33:21.139027 6984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:33:21.139046 6984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:33:21.139063 6984 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:33:21.139078 6984 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:33:21.144481 6984 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34575,127.0.98.61:45951
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:45951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
W20260925 16:33:21.144666 6984 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:34575,127.0.98.61:45951'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260925 16:33:21.145481 6984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:33:21.146440 6984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:33:21.154480 6989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:21.154529 6990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:21.154480 6992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:21.154665 6984 server_base.cc:1034] running on GCE node
I20260925 16:33:21.155274 6984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:33:21.156297 6984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:33:21.157485 6984 hybrid_clock.cc:648] HybridClock initialized: now 1790354001157475 us; error 30 us; skew 500 ppm
Sep 25 16:33:21 dist-test-slave-65pf krb5kdc[6475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354001, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:33:21.160151 6984 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260925 16:33:21.161020 6984 webserver.cc:460] Webserver started at http://127.0.98.61:36663/ using document root <none> and password file <none>
I20260925 16:33:21.161552 6984 fs_manager.cc:359] Metadata directory not provided
I20260925 16:33:21.161638 6984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:33:21.161855 6984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:33:21.163858 6984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/master-1/data/instance:
uuid: "9fbd60470ceb46e2bb9f413b41a66ddb"
format_stamp: "Formatted at 2026-09-25 16:33:21 on dist-test-slave-65pf"
server_key: "440e7fba76af21f5a1d536c7c1216b76"
server_key_iv: "fdff286b6ca02e9bb88703eb45a1c735"
server_key_version: "encryptionkey@0"
I20260925 16:33:21.164439 6984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/master-1/wal/instance:
uuid: "9fbd60470ceb46e2bb9f413b41a66ddb"
format_stamp: "Formatted at 2026-09-25 16:33:21 on dist-test-slave-65pf"
server_key: "440e7fba76af21f5a1d536c7c1216b76"
server_key_iv: "fdff286b6ca02e9bb88703eb45a1c735"
server_key_version: "encryptionkey@0"
I20260925 16:33:21.168174 6984 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 16:33:21.170463 6999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:21.171649 6984 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:33:21.171787 6984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/master-1/wal
uuid: "9fbd60470ceb46e2bb9f413b41a66ddb"
format_stamp: "Formatted at 2026-09-25 16:33:21 on dist-test-slave-65pf"
server_key: "440e7fba76af21f5a1d536c7c1216b76"
server_key_iv: "fdff286b6ca02e9bb88703eb45a1c735"
server_key_version: "encryptionkey@0"
I20260925 16:33:21.171895 6984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:33:21.196142 6984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:33:21.199280 6984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:33:21.199481 6984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:33:21.206148 6984 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:45951
I20260925 16:33:21.206447 7051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:45951 every 8 connection(s)
I20260925 16:33:21.207135 6984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/master-1/data/info.pb
I20260925 16:33:21.209830 7052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:33:21.211126 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 6984
I20260925 16:33:21.213804 7052 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34575 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 16:33:21 dist-test-slave-65pf krb5kdc[6475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 25 16:33:21 dist-test-slave-65pf krb5kdc[6475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354001, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 16:33:21.224929 6559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:21.215501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34688 (local address 127.0.98.62:34575)
0925 16:33:21.215631 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:21.215635 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:21.215905 (+ 270us) server_negotiation.cc:407] Connection header received
0925 16:33:21.216074 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:21.216080 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:21.216127 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:21.216213 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:21.216828 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.217328 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.218191 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.218403 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.218515 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:21.221151 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:21.221157 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:21.221161 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:21.221190 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:21.222928 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:21.223516 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:21.223520 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:21.223521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:21.223613 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:21.223930 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:21.223936 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:21.223940 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:21.224170 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:21.224281 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:21.224662 (+ 381us) server_negotiation.cc:299] Negotiation successful
0925 16:33:21.224806 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 16:33:21.225540 7056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:21.215430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34575 (local address 127.0.0.1:34688)
0925 16:33:21.215791 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:21.215827 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:21.215960 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:21.216259 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:21.216264 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:21.216600 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:21.216709 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:21.216715 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.217451 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.217459 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:21.218075 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:21.218083 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.218542 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.218546 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:21.218670 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:21.219221 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:21.219236 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:21.220992 (+ 1756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:33:21.223056 (+ 2064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:21.223063 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:21.223068 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:21.223374 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:21.223715 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:21.223718 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:21.223720 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:21.223833 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:21.224280 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:33:21.224289 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:33:21.224412 (+ 123us) client_negotiation.cc:769] Sending connection context
0925 16:33:21.225147 (+ 735us) client_negotiation.cc:240] Negotiation successful
0925 16:33:21.225299 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"spinlock_wait_cycles":2304,"thread_start_us":137,"threads_started":1}
I20260925 16:33:21.225548 7055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:21.212529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41522 (local address 127.0.98.61:45951)
0925 16:33:21.212926 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:21.212941 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:21.212976 (+ 35us) server_negotiation.cc:407] Connection header received
0925 16:33:21.213585 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:21.213602 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:21.213935 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:21.214266 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:21.215074 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.216456 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.217302 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.217764 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.217929 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:21.220661 (+ 2732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:21.220669 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:21.220682 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:21.220707 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:21.222489 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:21.223034 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:21.223041 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:21.223050 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:21.223165 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:21.223457 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:21.223459 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:21.223460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:21.223837 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:21.224052 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:21.224368 (+ 316us) server_negotiation.cc:299] Negotiation successful
0925 16:33:21.224659 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":130,"threads_started":1}
I20260925 16:33:21.227012 7052 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45951 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 16:33:21 dist-test-slave-65pf krb5kdc[6475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354001, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260925 16:33:21.234784 7056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:21.227510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45951 (local address 127.0.0.1:41524)
0925 16:33:21.227605 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:21.227616 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:21.227697 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:21.227918 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:21.227920 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:21.228070 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:21.228146 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:21.228151 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.229265 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.229269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:21.229836 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:21.229844 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.230306 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.230308 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:21.230378 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:21.230831 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:21.230842 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:21.231993 (+ 1151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:33:21.233570 (+ 1577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:21.233572 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:21.233574 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:21.233739 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:21.233955 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:21.233958 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:21.233959 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:21.234030 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:21.234336 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:33:21.234338 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:33:21.234406 (+ 68us) client_negotiation.cc:769] Sending connection context
0925 16:33:21.234486 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 16:33:21.234620 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 16:33:21.234794 7055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:21.227639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41524 (local address 127.0.98.61:45951)
0925 16:33:21.227745 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:21.227750 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:21.227763 (+ 13us) server_negotiation.cc:407] Connection header received
0925 16:33:21.227799 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:21.227801 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:21.227829 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:21.227906 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:21.228257 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.229156 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.229987 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.230216 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.230293 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:21.232111 (+ 1818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:21.232114 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:21.232116 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:21.232135 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:21.233485 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:21.233815 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:21.233818 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:21.233819 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:21.233883 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:21.234108 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:21.234110 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:21.234111 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:21.234259 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:21.234331 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:21.234540 (+ 209us) server_negotiation.cc:299] Negotiation successful
0925 16:33:21.234682 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 16:33:21.237426 7052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb: Bootstrap starting.
I20260925 16:33:21.239893 7052 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb: Neither blocks nor log segments found. Creating new log.
I20260925 16:33:21.240614 7052 log.cc:824] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb: Log is configured to *not* fsync() on all Append() calls
I20260925 16:33:21.242420 7052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb: No bootstrap required, opened a new log
I20260925 16:33:21.246104 7052 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6772881645114340bec15e5a7e09e6e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34575 } } peers { permanent_uuid: "9fbd60470ceb46e2bb9f413b41a66ddb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45951 } }
I20260925 16:33:21.246356 7052 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:33:21.246412 7052 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fbd60470ceb46e2bb9f413b41a66ddb, State: Initialized, Role: FOLLOWER
I20260925 16:33:21.246883 7052 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6772881645114340bec15e5a7e09e6e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34575 } } peers { permanent_uuid: "9fbd60470ceb46e2bb9f413b41a66ddb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45951 } }
I20260925 16:33:21.247367 7058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6772881645114340bec15e5a7e09e6e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34575 } } peers { permanent_uuid: "9fbd60470ceb46e2bb9f413b41a66ddb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45951 } } }
I20260925 16:33:21.247507 7058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:33:21.248219 7052 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb [sys.catalog]: configured and running, proceeding with master startup.
W20260925 16:33:21.253065 7069 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 16:33:21.253147 7069 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 16:33:21.254827 6984 master_runner.cc:428] Detected that this master 9fbd60470ceb46e2bb9f413b41a66ddb is joining an existing cluster
I20260925 16:33:21.254885 6984 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:45951
I20260925 16:33:21.264874 7075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:21.257539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45951 (local address 127.0.0.1:41536)
0925 16:33:21.257922 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:21.257934 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:21.258055 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:21.258328 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:21.258331 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:21.258655 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:21.258817 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:21.258825 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.259593 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.259595 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:21.260139 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:21.260144 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.260576 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.260579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:21.260663 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:21.261174 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:21.261185 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:21.261504 (+ 319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:33:21.263309 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:21.263313 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:21.263316 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:21.263548 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:21.263846 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:21.263850 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:21.263851 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:21.263942 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:21.264299 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:33:21.264303 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:33:21.264394 (+ 91us) client_negotiation.cc:769] Sending connection context
0925 16:33:21.264512 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 16:33:21.264670 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"mutex_wait_us":17,"thread_start_us":95,"threads_started":1}
I20260925 16:33:21.264927 7055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:21.257592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41536 (local address 127.0.98.61:45951)
0925 16:33:21.257680 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:21.257684 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:21.258005 (+ 321us) server_negotiation.cc:407] Connection header received
0925 16:33:21.258139 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:21.258142 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:21.258174 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:21.258214 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:21.258928 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.259499 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.260256 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.260458 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.260570 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:21.261634 (+ 1064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:21.261637 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:21.261640 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:21.261666 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:21.263175 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:21.263660 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:21.263665 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:21.263667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:21.263744 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:21.264029 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:21.264032 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:21.264034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:21.264203 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:21.264279 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:21.264597 (+ 318us) server_negotiation.cc:299] Negotiation successful
0925 16:33:21.264742 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":25}
I20260925 16:33:21.266780 6559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:21.257224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34692 (local address 127.0.98.62:34575)
0925 16:33:21.257372 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:21.257376 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:21.257926 (+ 550us) server_negotiation.cc:407] Connection header received
0925 16:33:21.258107 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:21.258110 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:21.258159 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:21.258212 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:21.258978 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.259797 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.261021 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.261355 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.261433 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:21.262951 (+ 1518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:21.262955 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:21.262958 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:21.262983 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:21.264954 (+ 1971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:21.265477 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:21.265481 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:21.265482 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:21.265597 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:21.265902 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:21.265905 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:21.265907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:21.266125 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:21.266221 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:21.266474 (+ 253us) server_negotiation.cc:299] Negotiation successful
0925 16:33:21.266618 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 16:33:21.266796 7076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:21.257131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34575 (local address 127.0.0.1:34692)
0925 16:33:21.257844 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:21.257858 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:21.257994 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:21.258354 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:21.258359 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:21.258665 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:21.258844 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:21.258851 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.259940 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.259943 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:21.260885 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:21.260895 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.261594 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.261598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:21.261701 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:21.262296 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:21.262311 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:21.262793 (+ 482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:33:21.265088 (+ 2295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:21.265092 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:21.265094 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:21.265329 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:21.265700 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:21.265703 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:21.265705 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:21.265795 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:21.266209 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:33:21.266211 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:33:21.266293 (+ 82us) client_negotiation.cc:769] Sending connection context
0925 16:33:21.266410 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 16:33:21.266611 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"mutex_wait_us":21,"thread_start_us":83,"threads_started":1}
Sep 25 16:33:21 dist-test-slave-65pf krb5kdc[6475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354000, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260925 16:33:21.280799 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:21.271086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45951 (local address 127.0.0.1:41542)
0925 16:33:21.271792 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:21.271831 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:21.271939 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:21.272230 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:21.272235 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:21.272574 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:21.272680 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:21.272686 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.273657 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.273660 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:21.274277 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:21.274284 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.274825 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.274829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:21.274948 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:21.275499 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:21.275510 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:21.277056 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:33:21.278909 (+ 1853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:21.278919 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:21.278926 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:21.279227 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:21.279531 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:21.279534 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:21.279536 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:21.279640 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:21.280112 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:33:21.280119 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:33:21.280250 (+ 131us) client_negotiation.cc:769] Sending connection context
0925 16:33:21.280405 (+ 155us) client_negotiation.cc:240] Negotiation successful
0925 16:33:21.280618 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":82,"threads_started":1}
I20260925 16:33:21.280838 7055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:21.271148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41542 (local address 127.0.98.61:45951)
0925 16:33:21.271269 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:21.271273 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:21.271904 (+ 631us) server_negotiation.cc:407] Connection header received
0925 16:33:21.272058 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:21.272061 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:21.272112 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:21.272194 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:21.272787 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.273557 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.274399 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.274696 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.274804 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:21.277190 (+ 2386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:21.277194 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:21.277196 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:21.277218 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:21.278807 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:21.279335 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:21.279339 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:21.279341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:21.279425 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:21.279739 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:21.279745 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:21.279749 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:21.279991 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:21.280093 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:21.280479 (+ 386us) server_negotiation.cc:299] Negotiation successful
0925 16:33:21.280617 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 16:33:21.282675 6520 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:45951: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "9fbd60470ceb46e2bb9f413b41a66ddb"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 45951
}
attrs {
promote: true
}
}
dest_uuid: "6772881645114340bec15e5a7e09e6e5"
cas_config_opid_index: -1
I20260925 16:33:21.283355 6520 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6772881645114340bec15e5a7e09e6e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6772881645114340bec15e5a7e09e6e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34575 } } peers { permanent_uuid: "9fbd60470ceb46e2bb9f413b41a66ddb" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45951 } attrs { promote: true } }
I20260925 16:33:21.285151 7080 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6772881645114340bec15e5a7e09e6e5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 9fbd60470ceb46e2bb9f413b41a66ddb (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6772881645114340bec15e5a7e09e6e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34575 } } peers { permanent_uuid: "9fbd60470ceb46e2bb9f413b41a66ddb" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45951 } attrs { promote: true } } }
I20260925 16:33:21.285884 7026 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:33:21.286968 7080 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:45951
I20260925 16:33:21.287356 7026 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb [term 1 FOLLOWER]: Refusing update from remote peer 6772881645114340bec15e5a7e09e6e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260925 16:33:21.287443 6984 master.cc:537] Master@127.0.98.61:45951 shutting down...
I20260925 16:33:21.287436 7079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6772881645114340bec15e5a7e09e6e5 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6772881645114340bec15e5a7e09e6e5" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6772881645114340bec15e5a7e09e6e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34575 } } peers { permanent_uuid: "9fbd60470ceb46e2bb9f413b41a66ddb" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45951 } attrs { promote: true } } }
I20260925 16:33:21.287635 7079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6772881645114340bec15e5a7e09e6e5 [sys.catalog]: This master's current role is: LEADER
I20260925 16:33:21.288128 7080 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6772881645114340bec15e5a7e09e6e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fbd60470ceb46e2bb9f413b41a66ddb" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45951 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 16:33:21.289938 6507 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260925 16:33:21.291455 6507 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6772881645114340bec15e5a7e09e6e5 -> Peer 9fbd60470ceb46e2bb9f413b41a66ddb (127.0.98.61:45951): Couldn't send request to peer 9fbd60470ceb46e2bb9f413b41a66ddb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 16:33:21.291994 6984 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:33:21.292279 6984 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:33:21.292388 6984 tablet_replica.cc:330] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb: stopping tablet replica
I20260925 16:33:21.296816 6984 master.cc:552] Master@127.0.98.61:45951 shutdown complete.
I20260925 16:33:21.296870 6984 master_runner.cc:305] Clearing existing system tablet
I20260925 16:33:21.297127 6984 fs_manager.cc:359] Metadata directory not provided
I20260925 16:33:21.297183 6984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260925 16:33:21.298561 6984 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 16:33:21.299643 7084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:21.299870 6984 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:33:21.300019 6984 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 16:33:21.300063 6984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/master-1/wal
uuid: "9fbd60470ceb46e2bb9f413b41a66ddb"
format_stamp: "Formatted at 2026-09-25 16:33:21 on dist-test-slave-65pf"
server_key: "440e7fba76af21f5a1d536c7c1216b76"
server_key_iv: "fdff286b6ca02e9bb88703eb45a1c735"
server_key_version: "encryptionkey@0"
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
I20260925 16:33:21.302052 6984 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:33:21.305011 6984 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 16:33:21.305111 6984 log.cc:1196] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb: Deleting WAL directory at /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260925 16:33:21.305379 6984 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb: Deleting consensus metadata
I20260925 16:33:21.305585 6984 master_runner.cc:315] Copying system tablet from 127.0.98.62:34575
I20260925 16:33:21.306972 6984 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:34575
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260925 16:33:21.318068 6559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:21.309658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34698 (local address 127.0.98.62:34575)
0925 16:33:21.309787 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:21.309792 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:21.309888 (+ 96us) server_negotiation.cc:407] Connection header received
0925 16:33:21.310053 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:21.310057 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:21.310109 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:21.310209 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:21.310798 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.311667 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.312563 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.312961 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.313077 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:21.314603 (+ 1526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:21.314607 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:21.314611 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:21.314638 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:21.316283 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:21.316727 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:21.316731 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:21.316733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:21.316818 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:21.317123 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:21.317127 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:21.317128 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:21.317368 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:21.317508 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:21.317774 (+ 266us) server_negotiation.cc:299] Negotiation successful
0925 16:33:21.317887 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 16:33:21.318076 7091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:21.309527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34575 (local address 127.0.0.1:34698)
0925 16:33:21.309800 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:21.309817 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:21.309942 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:21.310243 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:21.310245 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:21.310413 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:21.310554 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:21.310560 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.311782 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.311785 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:21.312452 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:21.312461 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.313088 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.313091 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:21.313164 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:21.314080 (+ 916us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:21.314091 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:21.314403 (+ 312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:33:21.316383 (+ 1980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:21.316385 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:21.316387 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:21.316638 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:21.316908 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:21.316910 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:21.316912 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:21.317024 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:21.317495 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:33:21.317498 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:33:21.317608 (+ 110us) client_negotiation.cc:769] Sending connection context
0925 16:33:21.317719 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 16:33:21.317872 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":112,"threads_started":1}
I20260925 16:33:21.318073 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37381
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34575,127.0.98.61:45951,127.0.98.60:37381
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 16:33:21.319197 6540 tablet_copy_service.cc:140] P 6772881645114340bec15e5a7e09e6e5: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 9fbd60470ceb46e2bb9f413b41a66ddb ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:34698)
I20260925 16:33:21.319382 6540 tablet_copy_service.cc:161] P 6772881645114340bec15e5a7e09e6e5: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 9fbd60470ceb46e2bb9f413b41a66ddb at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:34698: session id = 9fbd60470ceb46e2bb9f413b41a66ddb-00000000000000000000000000000000
I20260925 16:33:21.321416 6540 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 6772881645114340bec15e5a7e09e6e5: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 16:33:21.322968 6984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 16:33:21.325022 6984 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb: tablet copy: Starting download of 0 data blocks...
I20260925 16:33:21.325232 6984 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb: tablet copy: Starting download of 1 WAL segments...
I20260925 16:33:21.326954 6984 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 16:33:21.328516 6540 tablet_copy_service.cc:342] P 6772881645114340bec15e5a7e09e6e5: Request end of tablet copy session 9fbd60470ceb46e2bb9f413b41a66ddb-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:34698
I20260925 16:33:21.328655 6540 tablet_copy_service.cc:433] P 6772881645114340bec15e5a7e09e6e5: ending tablet copy session 9fbd60470ceb46e2bb9f413b41a66ddb-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 9fbd60470ceb46e2bb9f413b41a66ddb
I20260925 16:33:21.335353 6984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:33:21.336870 7096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:21.336954 7093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:21.337074 7094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:21.337239 6984 server_base.cc:1034] running on GCE node
I20260925 16:33:21.337394 6984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:33:21.337677 6984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:33:21.338826 6984 hybrid_clock.cc:648] HybridClock initialized: now 1790354001338818 us; error 26 us; skew 500 ppm
Sep 25 16:33:21 dist-test-slave-65pf krb5kdc[6475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354001, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:33:21.340283 6984 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260925 16:33:21.340749 6984 webserver.cc:460] Webserver started at http://127.0.98.61:42529/ using document root <none> and password file <none>
I20260925 16:33:21.340947 6984 fs_manager.cc:359] Metadata directory not provided
I20260925 16:33:21.341004 6984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:33:21.342119 6984 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 16:33:21.343122 7103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:21.343366 6984 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 16:33:21.343444 6984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/master-1/wal
uuid: "9fbd60470ceb46e2bb9f413b41a66ddb"
format_stamp: "Formatted at 2026-09-25 16:33:21 on dist-test-slave-65pf"
server_key: "440e7fba76af21f5a1d536c7c1216b76"
server_key_iv: "fdff286b6ca02e9bb88703eb45a1c735"
server_key_version: "encryptionkey@0"
I20260925 16:33:21.343515 6984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:33:21.357029 6984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:33:21.359025 6984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:33:21.363837 6984 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:45951
I20260925 16:33:21.364374 6984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/master-1/data/info.pb
I20260925 16:33:21.364780 7155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:45951 every 8 connection(s)
I20260925 16:33:21.365998 7156 sys_catalog.cc:263] Verifying existing consensus state
I20260925 16:33:21.367038 7156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb: Bootstrap starting.
I20260925 16:33:21.374964 7156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 16:33:21.375267 7156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb: Bootstrap complete.
I20260925 16:33:21.375731 7156 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6772881645114340bec15e5a7e09e6e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34575 } } peers { permanent_uuid: "9fbd60470ceb46e2bb9f413b41a66ddb" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45951 } attrs { promote: true } }
I20260925 16:33:21.375866 7156 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9fbd60470ceb46e2bb9f413b41a66ddb, State: Initialized, Role: LEARNER
I20260925 16:33:21.375998 7156 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6772881645114340bec15e5a7e09e6e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34575 } } peers { permanent_uuid: "9fbd60470ceb46e2bb9f413b41a66ddb" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45951 } attrs { promote: true } }
I20260925 16:33:21.376315 7159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6772881645114340bec15e5a7e09e6e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34575 } } peers { permanent_uuid: "9fbd60470ceb46e2bb9f413b41a66ddb" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45951 } attrs { promote: true } } }
I20260925 16:33:21.376477 7156 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:33:21.376477 7159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb [sys.catalog]: This master's current role is: LEARNER
I20260925 16:33:21.379770 7170 catalog_manager.cc:1259] Loaded cluster ID: 05ab274724e944f197784d0f601636d0
I20260925 16:33:21.379855 7170 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb: loading cluster ID for follower catalog manager: success
I20260925 16:33:21.381309 7170 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb: acquiring CA information for follower catalog manager: success
I20260925 16:33:21.382357 7170 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 16:33:21.469935 7092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:33:21.470209 7092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:33:21.470250 7092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:33:21.474309 7092 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:33:21.474367 7092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:33:21.474387 7092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:33:21.474403 7092 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:33:21.474418 7092 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:33:21.479681 7092 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34575,127.0.98.61:45951,127.0.98.60:37381
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:33:21.480851 7092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:33:21.481884 7092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:33:21.490465 7176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:21.490475 7175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:21.490866 7178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:21.490959 7092 server_base.cc:1034] running on GCE node
I20260925 16:33:21.491535 7092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:33:21.492673 7092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:33:21.493849 7092 hybrid_clock.cc:648] HybridClock initialized: now 1790354001493835 us; error 27 us; skew 500 ppm
Sep 25 16:33:21 dist-test-slave-65pf krb5kdc[6475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354001, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:33:21.496701 7092 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260925 16:33:21.497548 7092 webserver.cc:460] Webserver started at http://127.0.98.60:39305/ using document root <none> and password file <none>
I20260925 16:33:21.498098 7092 fs_manager.cc:359] Metadata directory not provided
I20260925 16:33:21.498158 7092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:33:21.498391 7092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:33:21.500511 7092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/master-2/data/instance:
uuid: "fffc9052c3ea4c7da2a350f0f2ca6d75"
format_stamp: "Formatted at 2026-09-25 16:33:21 on dist-test-slave-65pf"
server_key: "cd01208c6b0a27a82ccf5f5cef298b86"
server_key_iv: "a08c0946f9874afc3cf657d89ccb7b13"
server_key_version: "encryptionkey@0"
I20260925 16:33:21.501063 7092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/master-2/wal/instance:
uuid: "fffc9052c3ea4c7da2a350f0f2ca6d75"
format_stamp: "Formatted at 2026-09-25 16:33:21 on dist-test-slave-65pf"
server_key: "cd01208c6b0a27a82ccf5f5cef298b86"
server_key_iv: "a08c0946f9874afc3cf657d89ccb7b13"
server_key_version: "encryptionkey@0"
I20260925 16:33:21.504832 7092 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 16:33:21.507193 7185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:21.508414 7092 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:33:21.508554 7092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/master-2/wal
uuid: "fffc9052c3ea4c7da2a350f0f2ca6d75"
format_stamp: "Formatted at 2026-09-25 16:33:21 on dist-test-slave-65pf"
server_key: "cd01208c6b0a27a82ccf5f5cef298b86"
server_key_iv: "a08c0946f9874afc3cf657d89ccb7b13"
server_key_version: "encryptionkey@0"
I20260925 16:33:21.508653 7092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:33:21.522734 7092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:33:21.525936 7092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:33:21.526124 7092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:33:21.535586 7092 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:37381
I20260925 16:33:21.535597 7237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:37381 every 8 connection(s)
I20260925 16:33:21.536547 7092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/master-2/data/info.pb
I20260925 16:33:21.539320 7238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:33:21.543310 7238 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34575 } has no permanent_uuid. Determining permanent_uuid...
I20260925 16:33:21.545732 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 7092
I20260925 16:33:21.548141 6711 heartbeater.cc:503] Master 127.0.98.62:34575 was elected leader, sending a full tablet report...
Sep 25 16:33:21 dist-test-slave-65pf krb5kdc[6475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354001, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 25 16:33:21 dist-test-slave-65pf krb5kdc[6475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260925 16:33:21.557343 6559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:21.545486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34700 (local address 127.0.98.62:34575)
0925 16:33:21.545664 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:21.545670 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:21.546476 (+ 806us) server_negotiation.cc:407] Connection header received
0925 16:33:21.547637 (+ 1161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:21.547640 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:21.547686 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:21.547753 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:21.548874 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.549603 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.550812 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.551018 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.551108 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:21.553679 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:21.553683 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:21.553685 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:21.553707 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:21.555296 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:21.555820 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:21.555824 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:21.555826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:21.555906 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:21.556246 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:21.556250 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:21.556252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:21.556438 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:21.556543 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:21.557038 (+ 495us) server_negotiation.cc:299] Negotiation successful
0925 16:33:21.557168 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 16:33:21.558297 7241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:21.546518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53478 (local address 127.0.98.60:37381)
0925 16:33:21.546871 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:21.546882 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:21.546917 (+ 35us) server_negotiation.cc:407] Connection header received
0925 16:33:21.547422 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:21.547444 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:21.547692 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:21.547878 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:21.548746 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.549676 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.550625 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.551062 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.551250 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:21.554021 (+ 2771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:21.554027 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:21.554037 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:21.554057 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:21.555611 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:21.556044 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:21.556057 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:21.556059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:21.556163 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:21.556454 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:21.556457 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:21.556459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:21.556805 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:21.557028 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:21.557193 (+ 165us) server_negotiation.cc:299] Negotiation successful
0925 16:33:21.557383 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":271,"spinlock_wait_cycles":896,"thread_start_us":149,"threads_started":1}
I20260925 16:33:21.558296 7240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:21.545830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34575 (local address 127.0.0.1:34700)
0925 16:33:21.546328 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:21.546380 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:21.547422 (+ 1042us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:21.547826 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:21.547831 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:21.548363 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:21.548731 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:21.548746 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.549733 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.549739 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:21.550707 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:21.550716 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.551185 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.551189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:21.551287 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:21.551961 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:21.551984 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:21.553483 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:33:21.555420 (+ 1937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:21.555429 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:21.555439 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:21.555713 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:21.555997 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:21.556001 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:21.556003 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:21.556148 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:21.556549 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:33:21.556559 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:33:21.556818 (+ 259us) client_negotiation.cc:769] Sending connection context
0925 16:33:21.556995 (+ 177us) client_negotiation.cc:240] Negotiation successful
0925 16:33:21.557324 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":4,"spinlock_wait_cycles":5632,"thread_start_us":158,"threads_started":1}
I20260925 16:33:21.559607 7238 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45951 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 16:33:21 dist-test-slave-65pf krb5kdc[6475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354001, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260925 16:33:21.567999 7242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:21.560273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41554 (local address 127.0.98.61:45951)
0925 16:33:21.560613 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:21.560617 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:21.560631 (+ 14us) server_negotiation.cc:407] Connection header received
0925 16:33:21.560666 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:21.560668 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:21.560705 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:21.560776 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:21.561128 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.561831 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.562737 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.562952 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.563032 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:21.564805 (+ 1773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:21.564808 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:21.564810 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:21.564832 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:21.566418 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:21.566850 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:21.566853 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:21.566854 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:21.566929 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:21.567199 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:21.567201 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:21.567202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:21.567361 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:21.567468 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:21.567733 (+ 265us) server_negotiation.cc:299] Negotiation successful
0925 16:33:21.567852 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":138,"threads_started":1}
I20260925 16:33:21.567996 7240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:21.560092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45951 (local address 127.0.0.1:41554)
0925 16:33:21.560187 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:21.560203 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:21.560275 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:21.560805 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:21.560808 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:21.560936 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:21.561029 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:21.561033 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.561940 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.561944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:21.562631 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:21.562647 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.563040 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.563044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:21.563116 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:21.563547 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:21.563557 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:21.564680 (+ 1123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:33:21.566520 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:21.566523 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:21.566524 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:21.566768 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:21.567005 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:21.567007 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:21.567008 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:21.567118 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:21.567460 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:33:21.567463 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:33:21.567560 (+ 97us) client_negotiation.cc:769] Sending connection context
0925 16:33:21.567674 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 16:33:21.567814 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260925 16:33:21.568791 7238 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37381 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 16:33:21 dist-test-slave-65pf krb5kdc[6475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354001, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260925 16:33:21.575850 7240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:21.569250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37381 (local address 127.0.0.1:53484)
0925 16:33:21.569362 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:21.569373 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:21.569432 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:21.569639 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:21.569641 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:21.569765 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:21.569832 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:21.569836 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.570494 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.570496 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:21.571063 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:21.571070 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.571464 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.571466 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:21.571522 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:21.571914 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:21.571924 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:21.573064 (+ 1140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:33:21.574690 (+ 1626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:21.574692 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:21.574694 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:21.574861 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:21.575094 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:21.575096 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:21.575098 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:21.575162 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:21.575485 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:33:21.575487 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:33:21.575553 (+ 66us) client_negotiation.cc:769] Sending connection context
0925 16:33:21.575628 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 16:33:21.575713 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260925 16:33:21.575881 7241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:21.569361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53484 (local address 127.0.98.60:37381)
0925 16:33:21.569466 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:21.569470 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:21.569489 (+ 19us) server_negotiation.cc:407] Connection header received
0925 16:33:21.569517 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:21.569519 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:21.569552 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:21.569615 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:21.569921 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.570409 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.571153 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.571374 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.571452 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:21.573177 (+ 1725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:21.573180 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:21.573182 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:21.573201 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:21.574553 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:21.574939 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:21.574941 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:21.574942 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:21.575009 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:21.575233 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:21.575235 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:21.575236 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:21.575408 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:21.575481 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:21.575680 (+ 199us) server_negotiation.cc:299] Negotiation successful
0925 16:33:21.575766 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 16:33:21.579263 7238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fffc9052c3ea4c7da2a350f0f2ca6d75: Bootstrap starting.
I20260925 16:33:21.581920 7238 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fffc9052c3ea4c7da2a350f0f2ca6d75: Neither blocks nor log segments found. Creating new log.
I20260925 16:33:21.582700 7238 log.cc:824] T 00000000000000000000000000000000 P fffc9052c3ea4c7da2a350f0f2ca6d75: Log is configured to *not* fsync() on all Append() calls
I20260925 16:33:21.584532 7238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fffc9052c3ea4c7da2a350f0f2ca6d75: No bootstrap required, opened a new log
I20260925 16:33:21.587531 7238 raft_consensus.cc:348] T 00000000000000000000000000000000 P fffc9052c3ea4c7da2a350f0f2ca6d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6772881645114340bec15e5a7e09e6e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34575 } } peers { permanent_uuid: "9fbd60470ceb46e2bb9f413b41a66ddb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45951 } } peers { permanent_uuid: "fffc9052c3ea4c7da2a350f0f2ca6d75" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37381 } }
I20260925 16:33:21.587769 7238 raft_consensus.cc:374] T 00000000000000000000000000000000 P fffc9052c3ea4c7da2a350f0f2ca6d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:33:21.587846 7238 raft_consensus.cc:729] T 00000000000000000000000000000000 P fffc9052c3ea4c7da2a350f0f2ca6d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fffc9052c3ea4c7da2a350f0f2ca6d75, State: Initialized, Role: FOLLOWER
I20260925 16:33:21.588272 7238 consensus_queue.cc:260] T 00000000000000000000000000000000 P fffc9052c3ea4c7da2a350f0f2ca6d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6772881645114340bec15e5a7e09e6e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34575 } } peers { permanent_uuid: "9fbd60470ceb46e2bb9f413b41a66ddb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45951 } } peers { permanent_uuid: "fffc9052c3ea4c7da2a350f0f2ca6d75" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37381 } }
I20260925 16:33:21.588708 7244 sys_catalog.cc:455] T 00000000000000000000000000000000 P fffc9052c3ea4c7da2a350f0f2ca6d75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6772881645114340bec15e5a7e09e6e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34575 } } peers { permanent_uuid: "9fbd60470ceb46e2bb9f413b41a66ddb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45951 } } peers { permanent_uuid: "fffc9052c3ea4c7da2a350f0f2ca6d75" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37381 } } }
I20260925 16:33:21.588819 7244 sys_catalog.cc:458] T 00000000000000000000000000000000 P fffc9052c3ea4c7da2a350f0f2ca6d75 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:33:21.589382 7238 sys_catalog.cc:564] T 00000000000000000000000000000000 P fffc9052c3ea4c7da2a350f0f2ca6d75 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 16:33:21.594413 7255 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fffc9052c3ea4c7da2a350f0f2ca6d75: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 16:33:21.594496 7255 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 16:33:21.597836 7092 master_runner.cc:428] Detected that this master fffc9052c3ea4c7da2a350f0f2ca6d75 is joining an existing cluster
I20260925 16:33:21.597918 7092 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:37381
I20260925 16:33:21.608506 7261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:21.599992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45951 (local address 127.0.0.1:41556)
0925 16:33:21.600309 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:21.600324 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:21.600452 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:21.600810 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:21.600813 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:21.601091 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:21.601338 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:21.601343 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.602658 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.602661 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:21.603509 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:21.603519 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.604185 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.604188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:21.604380 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:21.604970 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:21.604981 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:21.605336 (+ 355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:33:21.607198 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:21.607200 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:21.607202 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:21.607387 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:21.607654 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:21.607656 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:21.607657 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:21.607727 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:21.608112 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:33:21.608115 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:33:21.608185 (+ 70us) client_negotiation.cc:769] Sending connection context
0925 16:33:21.608264 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 16:33:21.608361 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"mutex_wait_us":144,"spinlock_wait_cycles":5888,"thread_start_us":94,"threads_started":1}
I20260925 16:33:21.608608 7242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:21.599971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41556 (local address 127.0.98.61:45951)
0925 16:33:21.600096 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:21.600100 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:21.600381 (+ 281us) server_negotiation.cc:407] Connection header received
0925 16:33:21.600572 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:21.600574 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:21.600619 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:21.600671 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:21.601469 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.602297 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.603635 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.603968 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.604031 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:21.605471 (+ 1440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:21.605474 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:21.605477 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:21.605504 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:21.607072 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:21.607474 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:21.607478 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:21.607480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:21.607567 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:21.607809 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:21.607812 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:21.607814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:21.608026 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:21.608121 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:21.608333 (+ 212us) server_negotiation.cc:299] Negotiation successful
0925 16:33:21.608453 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 16:33:21.611613 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 6582
I20260925 16:33:21.617630 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:43207
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:34575
--webserver_port=33421
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:34575,127.0.98.61:45951
--tserver_master_addrs=127.0.98.62:34575,127.0.98.61:45951,127.0.98.60:37381 with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 16:33:21.623844 7241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:21.600549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53500 (local address 127.0.98.60:37381)
0925 16:33:21.600810 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:21.600814 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:21.600825 (+ 11us) server_negotiation.cc:407] Connection header received
0925 16:33:21.600920 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:21.600923 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:21.600957 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:21.601015 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:21.601597 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.602343 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.604121 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.604461 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.604576 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:21.619788 (+ 15212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:21.619793 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:21.619798 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:21.619829 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:21.621989 (+ 2160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:21.622454 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:21.622457 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:21.622459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:21.622525 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:21.622920 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:21.622922 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:21.622924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:21.623152 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:21.623221 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:21.623527 (+ 306us) server_negotiation.cc:299] Negotiation successful
0925 16:33:21.623654 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":199,"spinlock_wait_cycles":896}
I20260925 16:33:21.624475 7263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:21.600300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37381 (local address 127.0.0.1:53500)
0925 16:33:21.600715 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:21.600729 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:21.600821 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:21.601151 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:21.601154 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:21.601371 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:21.601483 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:21.601490 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.602464 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.602467 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:21.603998 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:21.604008 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.604578 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.604582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:21.604688 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:21.619154 (+ 14466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:21.619171 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:21.619617 (+ 446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:33:21.622113 (+ 2496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:21.622116 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:21.622118 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:21.622350 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:21.622673 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:21.622675 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:21.622677 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:21.622819 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:21.623276 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:33:21.623281 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:33:21.623368 (+ 87us) client_negotiation.cc:769] Sending connection context
0925 16:33:21.624074 (+ 706us) client_negotiation.cc:240] Negotiation successful
0925 16:33:21.624301 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"mutex_wait_us":111,"spinlock_wait_cycles":1408,"thread_start_us":70,"threads_started":1}
I20260925 16:33:21.624547 6559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:21.599978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34714 (local address 127.0.98.62:34575)
0925 16:33:21.600121 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:21.600126 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:21.600374 (+ 248us) server_negotiation.cc:407] Connection header received
0925 16:33:21.600566 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:21.600569 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:21.600619 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:21.600671 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:21.601469 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.602286 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.603611 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.603941 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.604018 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:21.620471 (+ 16453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:21.620474 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:21.620477 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:21.620503 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:21.622528 (+ 2025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:21.623059 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:21.623062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:21.623064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:21.623139 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:21.623435 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:21.623442 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:21.623445 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:21.623654 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:21.623724 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:21.624284 (+ 560us) server_negotiation.cc:299] Negotiation successful
0925 16:33:21.624394 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 16:33:21.625205 7262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:21.599871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34575 (local address 127.0.0.1:34714)
0925 16:33:21.600290 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:21.600304 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:21.600452 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:21.600891 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:21.600893 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:21.601100 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:21.601335 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:21.601343 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.602426 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.602432 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:21.603474 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:21.603484 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.604186 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.604188 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:21.604784 (+ 596us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:21.619798 (+ 15014us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:21.619811 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:21.620317 (+ 506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:33:21.622675 (+ 2358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:21.622677 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:21.622679 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:21.622942 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:21.623251 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:21.623255 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:21.623257 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:21.623332 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:21.623791 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:33:21.623794 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:33:21.623892 (+ 98us) client_negotiation.cc:769] Sending connection context
0925 16:33:21.624907 (+ 1015us) client_negotiation.cc:240] Negotiation successful
0925 16:33:21.625052 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":96,"spinlock_wait_cycles":19712,"thread_start_us":115,"threads_started":1}
Sep 25 16:33:21 dist-test-slave-65pf krb5kdc[6475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354000, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260925 16:33:21.636595 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:21.628922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37381 (local address 127.0.0.1:53510)
0925 16:33:21.629092 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:21.629105 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:21.629192 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:21.629598 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:21.629603 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:21.629761 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:21.629887 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:21.629894 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.630619 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.630625 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:21.631169 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:21.631177 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.631622 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.631625 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:21.631715 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:21.632180 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:21.632193 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:21.633459 (+ 1266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:33:21.635163 (+ 1704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:21.635166 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:21.635169 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:21.635359 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:21.635640 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:21.635645 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:21.635647 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:21.635732 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:21.636097 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:33:21.636100 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:33:21.636197 (+ 97us) client_negotiation.cc:769] Sending connection context
0925 16:33:21.636300 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 16:33:21.636439 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97}
I20260925 16:33:21.636593 7241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:21.628960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53510 (local address 127.0.98.60:37381)
0925 16:33:21.629055 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:21.629059 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:21.629167 (+ 108us) server_negotiation.cc:407] Connection header received
0925 16:33:21.629302 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:21.629304 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:21.629396 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:21.629463 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:21.629998 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.630487 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.631285 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.631513 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.631621 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:21.633594 (+ 1973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:21.633598 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:21.633600 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:21.633622 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:21.635035 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:21.635458 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:21.635463 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:21.635465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:21.635547 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:21.635820 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:21.635826 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:21.635828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:21.635987 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:21.636092 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:21.636347 (+ 255us) server_negotiation.cc:299] Negotiation successful
0925 16:33:21.636439 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 16:33:21.637856 6520 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:37381: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "fffc9052c3ea4c7da2a350f0f2ca6d75"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 37381
}
attrs {
promote: true
}
}
dest_uuid: "6772881645114340bec15e5a7e09e6e5"
cas_config_opid_index: 5
I20260925 16:33:21.638438 6520 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6772881645114340bec15e5a7e09e6e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6772881645114340bec15e5a7e09e6e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34575 } } peers { permanent_uuid: "9fbd60470ceb46e2bb9f413b41a66ddb" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45951 } attrs { promote: true } } peers { permanent_uuid: "fffc9052c3ea4c7da2a350f0f2ca6d75" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37381 } attrs { promote: true } }
I20260925 16:33:21.639657 7080 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6772881645114340bec15e5a7e09e6e5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER fffc9052c3ea4c7da2a350f0f2ca6d75 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6772881645114340bec15e5a7e09e6e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34575 } } peers { permanent_uuid: "9fbd60470ceb46e2bb9f413b41a66ddb" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45951 } attrs { promote: true } } peers { permanent_uuid: "fffc9052c3ea4c7da2a350f0f2ca6d75" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37381 } attrs { promote: true } } }
I20260925 16:33:21.640930 7080 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:37381
I20260925 16:33:21.641108 7080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6772881645114340bec15e5a7e09e6e5 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6772881645114340bec15e5a7e09e6e5" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6772881645114340bec15e5a7e09e6e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34575 } } peers { permanent_uuid: "9fbd60470ceb46e2bb9f413b41a66ddb" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45951 } attrs { promote: true } } peers { permanent_uuid: "fffc9052c3ea4c7da2a350f0f2ca6d75" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37381 } attrs { promote: true } } }
I20260925 16:33:21.641218 7080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6772881645114340bec15e5a7e09e6e5 [sys.catalog]: This master's current role is: LEADER
I20260925 16:33:21.641737 7092 master.cc:537] Master@127.0.98.60:37381 shutting down...
I20260925 16:33:21.642099 7212 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fffc9052c3ea4c7da2a350f0f2ca6d75 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:33:21.643705 7212 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fffc9052c3ea4c7da2a350f0f2ca6d75 [term 1 FOLLOWER]: Refusing update from remote peer 6772881645114340bec15e5a7e09e6e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 16:33:21.645342 7079 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6772881645114340bec15e5a7e09e6e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fffc9052c3ea4c7da2a350f0f2ca6d75" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37381 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 16:33:21.647027 6504 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6772881645114340bec15e5a7e09e6e5 -> Peer fffc9052c3ea4c7da2a350f0f2ca6d75 (127.0.98.60:37381): Couldn't send request to peer fffc9052c3ea4c7da2a350f0f2ca6d75. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 16:33:21.649331 7092 raft_consensus.cc:2229] T 00000000000000000000000000000000 P fffc9052c3ea4c7da2a350f0f2ca6d75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:33:21.649689 7092 raft_consensus.cc:2258] T 00000000000000000000000000000000 P fffc9052c3ea4c7da2a350f0f2ca6d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:33:21.649987 7092 tablet_replica.cc:330] T 00000000000000000000000000000000 P fffc9052c3ea4c7da2a350f0f2ca6d75: stopping tablet replica
I20260925 16:33:21.653662 7242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:21.640031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41560 (local address 127.0.98.61:45951)
0925 16:33:21.640149 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:21.640154 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:21.648587 (+ 8433us) server_negotiation.cc:407] Connection header received
0925 16:33:21.649015 (+ 428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:21.649021 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:21.649104 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:21.649178 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:33:21.649602 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.650569 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.652053 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.652738 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.652834 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:21.653009 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:21.653453 (+ 444us) server_negotiation.cc:299] Negotiation successful
0925 16:33:21.653536 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 16:33:21.654402 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:21.639783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45951 (local address 127.0.0.1:41560)
0925 16:33:21.648505 (+ 8722us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:21.648523 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:21.648864 (+ 341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:21.649358 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:21.649361 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:21.649379 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:33:21.649474 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:21.649480 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.650736 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.650742 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:21.651895 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:21.651908 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.653183 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.653186 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:21.653276 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:21.653329 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 16:33:21.654004 (+ 675us) client_negotiation.cc:240] Negotiation successful
0925 16:33:21.654091 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8641}
I20260925 16:33:21.654963 7130 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb [term 1 LEARNER]: Refusing update from remote peer 6772881645114340bec15e5a7e09e6e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 16:33:21.655347 7092 master.cc:552] Master@127.0.98.60:37381 shutdown complete.
I20260925 16:33:21.655431 7092 master_runner.cc:305] Clearing existing system tablet
I20260925 16:33:21.655418 7079 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6772881645114340bec15e5a7e09e6e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fbd60470ceb46e2bb9f413b41a66ddb" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45951 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260925 16:33:21.655797 7092 fs_manager.cc:359] Metadata directory not provided
I20260925 16:33:21.655898 7092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:33:21.657292 7130 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER fffc9052c3ea4c7da2a350f0f2ca6d75 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6772881645114340bec15e5a7e09e6e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34575 } } peers { permanent_uuid: "9fbd60470ceb46e2bb9f413b41a66ddb" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45951 } attrs { promote: true } } peers { permanent_uuid: "fffc9052c3ea4c7da2a350f0f2ca6d75" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37381 } attrs { promote: true } } }
I20260925 16:33:21.658949 7159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6772881645114340bec15e5a7e09e6e5. Latest consensus state: current_term: 1 leader_uuid: "6772881645114340bec15e5a7e09e6e5" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6772881645114340bec15e5a7e09e6e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34575 } } peers { permanent_uuid: "9fbd60470ceb46e2bb9f413b41a66ddb" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45951 } attrs { promote: true } } peers { permanent_uuid: "fffc9052c3ea4c7da2a350f0f2ca6d75" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37381 } attrs { promote: true } } }
I20260925 16:33:21.659068 7159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb [sys.catalog]: This master's current role is: LEARNER
I20260925 16:33:21.659166 7159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6772881645114340bec15e5a7e09e6e5" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6772881645114340bec15e5a7e09e6e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34575 } } peers { permanent_uuid: "9fbd60470ceb46e2bb9f413b41a66ddb" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45951 } attrs { promote: true } } peers { permanent_uuid: "fffc9052c3ea4c7da2a350f0f2ca6d75" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37381 } attrs { promote: true } } }
I20260925 16:33:21.659260 7159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb [sys.catalog]: This master's current role is: LEARNER
I20260925 16:33:21.659579 7079 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 6772881645114340bec15e5a7e09e6e5: attempting to promote NON_VOTER 9fbd60470ceb46e2bb9f413b41a66ddb to VOTER
I20260925 16:33:21.659981 7079 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6772881645114340bec15e5a7e09e6e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6772881645114340bec15e5a7e09e6e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34575 } } peers { permanent_uuid: "9fbd60470ceb46e2bb9f413b41a66ddb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45951 } attrs { promote: false } } peers { permanent_uuid: "fffc9052c3ea4c7da2a350f0f2ca6d75" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37381 } attrs { promote: true } }
I20260925 16:33:21.661331 7130 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb [term 1 LEARNER]: Refusing update from remote peer 6772881645114340bec15e5a7e09e6e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260925 16:33:21.661484 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:21.660940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37381 (local address 127.0.0.1:53524)
0925 16:33:21.661289 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:21.661359 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281}
I20260925 16:33:21.661696 7268 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6772881645114340bec15e5a7e09e6e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fbd60470ceb46e2bb9f413b41a66ddb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45951 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260925 16:33:21.662374 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:21.662030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37381 (local address 127.0.0.1:53536)
0925 16:33:21.662256 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:21.662304 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":76,"threads_started":1}
I20260925 16:33:21.662669 7092 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.001s sys 0.000s
W20260925 16:33:21.662806 6504 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6772881645114340bec15e5a7e09e6e5 -> Peer fffc9052c3ea4c7da2a350f0f2ca6d75 (127.0.98.60:37381): Couldn't send request to peer fffc9052c3ea4c7da2a350f0f2ca6d75. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37381: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 16:33:21.663662 7270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:21.663741 7080 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6772881645114340bec15e5a7e09e6e5 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9fbd60470ceb46e2bb9f413b41a66ddb (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6772881645114340bec15e5a7e09e6e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34575 } } peers { permanent_uuid: "9fbd60470ceb46e2bb9f413b41a66ddb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45951 } attrs { promote: false } } peers { permanent_uuid: "fffc9052c3ea4c7da2a350f0f2ca6d75" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37381 } attrs { promote: true } } }
I20260925 16:33:21.663883 7092 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:33:21.664206 7092 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 16:33:21.664283 7092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/master-2/wal
uuid: "fffc9052c3ea4c7da2a350f0f2ca6d75"
format_stamp: "Formatted at 2026-09-25 16:33:21 on dist-test-slave-65pf"
server_key: "cd01208c6b0a27a82ccf5f5cef298b86"
server_key_iv: "a08c0946f9874afc3cf657d89ccb7b13"
server_key_version: "encryptionkey@0"
I20260925 16:33:21.665268 7080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6772881645114340bec15e5a7e09e6e5 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6772881645114340bec15e5a7e09e6e5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6772881645114340bec15e5a7e09e6e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34575 } } peers { permanent_uuid: "9fbd60470ceb46e2bb9f413b41a66ddb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45951 } attrs { promote: false } } peers { permanent_uuid: "fffc9052c3ea4c7da2a350f0f2ca6d75" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37381 } attrs { promote: true } } }
I20260925 16:33:21.665378 7080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6772881645114340bec15e5a7e09e6e5 [sys.catalog]: This master's current role is: LEADER
I20260925 16:33:21.665913 7079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6772881645114340bec15e5a7e09e6e5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6772881645114340bec15e5a7e09e6e5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6772881645114340bec15e5a7e09e6e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34575 } } peers { permanent_uuid: "9fbd60470ceb46e2bb9f413b41a66ddb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45951 } attrs { promote: false } } peers { permanent_uuid: "fffc9052c3ea4c7da2a350f0f2ca6d75" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37381 } attrs { promote: true } } }
I20260925 16:33:21.666033 7079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6772881645114340bec15e5a7e09e6e5 [sys.catalog]: This master's current role is: LEADER
I20260925 16:33:21.666705 7130 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9fbd60470ceb46e2bb9f413b41a66ddb (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6772881645114340bec15e5a7e09e6e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34575 } } peers { permanent_uuid: "9fbd60470ceb46e2bb9f413b41a66ddb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45951 } attrs { promote: false } } peers { permanent_uuid: "fffc9052c3ea4c7da2a350f0f2ca6d75" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37381 } attrs { promote: true } } }
I20260925 16:33:21.666805 7092 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P fffc9052c3ea4c7da2a350f0f2ca6d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:33:21.668159 7267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6772881645114340bec15e5a7e09e6e5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6772881645114340bec15e5a7e09e6e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34575 } } peers { permanent_uuid: "9fbd60470ceb46e2bb9f413b41a66ddb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45951 } attrs { promote: false } } peers { permanent_uuid: "fffc9052c3ea4c7da2a350f0f2ca6d75" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37381 } attrs { promote: true } } }
I20260925 16:33:21.668288 7267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:33:21.669243 7092 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P fffc9052c3ea4c7da2a350f0f2ca6d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 16:33:21.669353 7092 log.cc:1196] T 00000000000000000000000000000000 P fffc9052c3ea4c7da2a350f0f2ca6d75: Deleting WAL directory at /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260925 16:33:21.669607 7092 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P fffc9052c3ea4c7da2a350f0f2ca6d75: Deleting consensus metadata
I20260925 16:33:21.669778 7092 master_runner.cc:315] Copying system tablet from 127.0.98.62:34575
I20260925 16:33:21.670892 7092 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P fffc9052c3ea4c7da2a350f0f2ca6d75: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:34575
I20260925 16:33:21.682950 6559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:21.671400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34730 (local address 127.0.98.62:34575)
0925 16:33:21.671507 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:21.671511 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:21.676037 (+ 4526us) server_negotiation.cc:407] Connection header received
0925 16:33:21.676207 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:21.676210 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:21.676256 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:21.676340 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:21.676771 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.677349 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.678316 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.678574 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.678700 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:21.679834 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:21.679837 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:21.679840 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:21.679863 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:21.681307 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:21.681727 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:21.681732 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:21.681734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:21.681811 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:21.682089 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:21.682092 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:21.682094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:21.682292 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:21.682395 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:21.682662 (+ 267us) server_negotiation.cc:299] Negotiation successful
0925 16:33:21.682780 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 16:33:21.682967 7277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:21.671364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34575 (local address 127.0.0.1:34730)
0925 16:33:21.675944 (+ 4580us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:21.675958 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:21.676096 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:21.676356 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:21.676359 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:21.676564 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:21.676656 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:21.676661 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.677492 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.677496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:21.678189 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:21.678198 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.678704 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.678708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:21.678807 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:21.679357 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:21.679377 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:21.679694 (+ 317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:33:21.681431 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:21.681434 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:21.681436 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:21.681631 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:21.681896 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:21.681901 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:21.681903 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:21.681985 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:21.682385 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:33:21.682388 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:33:21.682485 (+ 97us) client_negotiation.cc:769] Sending connection context
0925 16:33:21.682619 (+ 134us) client_negotiation.cc:240] Negotiation successful
0925 16:33:21.682781 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4505,"thread_start_us":69,"threads_started":1}
I20260925 16:33:21.683754 6540 tablet_copy_service.cc:140] P 6772881645114340bec15e5a7e09e6e5: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer fffc9052c3ea4c7da2a350f0f2ca6d75 ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:34730)
I20260925 16:33:21.683867 6540 tablet_copy_service.cc:161] P 6772881645114340bec15e5a7e09e6e5: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer fffc9052c3ea4c7da2a350f0f2ca6d75 at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:34730: session id = fffc9052c3ea4c7da2a350f0f2ca6d75-00000000000000000000000000000000
I20260925 16:33:21.684690 6540 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 6772881645114340bec15e5a7e09e6e5: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 16:33:21.685774 7092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 16:33:21.687811 7092 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P fffc9052c3ea4c7da2a350f0f2ca6d75: tablet copy: Starting download of 0 data blocks...
I20260925 16:33:21.688009 7092 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P fffc9052c3ea4c7da2a350f0f2ca6d75: tablet copy: Starting download of 1 WAL segments...
I20260925 16:33:21.689397 7092 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P fffc9052c3ea4c7da2a350f0f2ca6d75: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 16:33:21.691147 6540 tablet_copy_service.cc:342] P 6772881645114340bec15e5a7e09e6e5: Request end of tablet copy session fffc9052c3ea4c7da2a350f0f2ca6d75-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:34730
I20260925 16:33:21.691226 6540 tablet_copy_service.cc:433] P 6772881645114340bec15e5a7e09e6e5: ending tablet copy session fffc9052c3ea4c7da2a350f0f2ca6d75-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer fffc9052c3ea4c7da2a350f0f2ca6d75
I20260925 16:33:21.697577 7092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:33:21.699271 7281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:21.699308 7279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:21.703233 7278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:21.703500 7092 server_base.cc:1034] running on GCE node
I20260925 16:33:21.703658 7092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:33:21.703979 7092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:33:21.705113 7092 hybrid_clock.cc:648] HybridClock initialized: now 1790354001705102 us; error 30 us; skew 500 ppm
Sep 25 16:33:21 dist-test-slave-65pf krb5kdc[6475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354001, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:33:21.706645 7092 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260925 16:33:21.707113 7092 webserver.cc:460] Webserver started at http://127.0.98.60:42543/ using document root <none> and password file <none>
I20260925 16:33:21.707327 7092 fs_manager.cc:359] Metadata directory not provided
I20260925 16:33:21.707382 7092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:33:21.708452 7092 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 16:33:21.709378 7288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:21.709627 7092 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 16:33:21.709720 7092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/master-2/wal
uuid: "fffc9052c3ea4c7da2a350f0f2ca6d75"
format_stamp: "Formatted at 2026-09-25 16:33:21 on dist-test-slave-65pf"
server_key: "cd01208c6b0a27a82ccf5f5cef298b86"
server_key_iv: "a08c0946f9874afc3cf657d89ccb7b13"
server_key_version: "encryptionkey@0"
I20260925 16:33:21.709805 7092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:33:21.716975 7092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:33:21.718983 7092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:33:21.724337 7092 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:37381
I20260925 16:33:21.724872 7092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/master-2/data/info.pb
I20260925 16:33:21.725379 7340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:37381 every 8 connection(s)
I20260925 16:33:21.726419 7341 sys_catalog.cc:263] Verifying existing consensus state
I20260925 16:33:21.727303 7341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fffc9052c3ea4c7da2a350f0f2ca6d75: Bootstrap starting.
I20260925 16:33:21.736084 7341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fffc9052c3ea4c7da2a350f0f2ca6d75: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 16:33:21.736379 7341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fffc9052c3ea4c7da2a350f0f2ca6d75: Bootstrap complete.
I20260925 16:33:21.736865 7341 raft_consensus.cc:348] T 00000000000000000000000000000000 P fffc9052c3ea4c7da2a350f0f2ca6d75 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6772881645114340bec15e5a7e09e6e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34575 } } peers { permanent_uuid: "9fbd60470ceb46e2bb9f413b41a66ddb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45951 } attrs { promote: false } } peers { permanent_uuid: "fffc9052c3ea4c7da2a350f0f2ca6d75" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37381 } attrs { promote: true } }
I20260925 16:33:21.737000 7341 raft_consensus.cc:729] T 00000000000000000000000000000000 P fffc9052c3ea4c7da2a350f0f2ca6d75 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fffc9052c3ea4c7da2a350f0f2ca6d75, State: Initialized, Role: LEARNER
I20260925 16:33:21.737133 7341 consensus_queue.cc:260] T 00000000000000000000000000000000 P fffc9052c3ea4c7da2a350f0f2ca6d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6772881645114340bec15e5a7e09e6e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34575 } } peers { permanent_uuid: "9fbd60470ceb46e2bb9f413b41a66ddb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45951 } attrs { promote: false } } peers { permanent_uuid: "fffc9052c3ea4c7da2a350f0f2ca6d75" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37381 } attrs { promote: true } }
I20260925 16:33:21.737447 7344 sys_catalog.cc:455] T 00000000000000000000000000000000 P fffc9052c3ea4c7da2a350f0f2ca6d75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6772881645114340bec15e5a7e09e6e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34575 } } peers { permanent_uuid: "9fbd60470ceb46e2bb9f413b41a66ddb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45951 } attrs { promote: false } } peers { permanent_uuid: "fffc9052c3ea4c7da2a350f0f2ca6d75" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37381 } attrs { promote: true } } }
I20260925 16:33:21.737609 7341 sys_catalog.cc:564] T 00000000000000000000000000000000 P fffc9052c3ea4c7da2a350f0f2ca6d75 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:33:21.737622 7344 sys_catalog.cc:458] T 00000000000000000000000000000000 P fffc9052c3ea4c7da2a350f0f2ca6d75 [sys.catalog]: This master's current role is: LEARNER
I20260925 16:33:21.741398 7355 catalog_manager.cc:1259] Loaded cluster ID: 05ab274724e944f197784d0f601636d0
I20260925 16:33:21.741466 7355 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fffc9052c3ea4c7da2a350f0f2ca6d75: loading cluster ID for follower catalog manager: success
I20260925 16:33:21.743183 7355 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fffc9052c3ea4c7da2a350f0f2ca6d75: acquiring CA information for follower catalog manager: success
I20260925 16:33:21.744653 7355 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fffc9052c3ea4c7da2a350f0f2ca6d75: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 16:33:21.772766 6845 heartbeater.cc:503] Master 127.0.98.62:34575 was elected leader, sending a full tablet report...
W20260925 16:33:21.773634 7264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:33:21.773943 7264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:33:21.774019 7264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:33:21.780138 7264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:33:21.780212 7264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:33:21.780298 7264 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 16:33:21.788192 7264 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:43207
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=33421
--tserver_master_addrs=127.0.98.62:34575,127.0.98.61:45951,127.0.98.60:37381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7264
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:33:21.789624 7264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:33:21.790907 7264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:33:21.801944 7363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:21.802043 7360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:21.801945 7361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:21.802452 7264 server_base.cc:1034] running on GCE node
I20260925 16:33:21.803123 7264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:33:21.803720 7264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:33:21.804909 7264 hybrid_clock.cc:648] HybridClock initialized: now 1790354001804872 us; error 50 us; skew 500 ppm
Sep 25 16:33:21 dist-test-slave-65pf krb5kdc[6475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354001, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:33:21.807500 7264 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260925 16:33:21.808393 7264 webserver.cc:460] Webserver started at http://127.0.98.1:33421/ using document root <none> and password file <none>
I20260925 16:33:21.808939 7264 fs_manager.cc:359] Metadata directory not provided
I20260925 16:33:21.809048 7264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:33:21.812881 7264 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260925 16:33:21.814865 7370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:21.816080 7264 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 16:33:21.816220 7264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-0/wal
uuid: "37e3504cf5f848cc912031815590b8a8"
format_stamp: "Formatted at 2026-09-25 16:33:20 on dist-test-slave-65pf"
server_key: "1ba7b1809b1f25b9be2f46962f76475f"
server_key_iv: "27c52fc80434bc5eb335ec37be52eddf"
server_key_version: "encryptionkey@0"
I20260925 16:33:21.816754 7264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:33:21.840852 7264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:33:21.843899 7264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:33:21.844125 7264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:33:21.844707 7264 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:33:21.845561 7264 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:33:21.845631 7264 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:21.845700 7264 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:33:21.845749 7264 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:21.855126 7264 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43207
I20260925 16:33:21.855154 7483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43207 every 8 connection(s)
I20260925 16:33:21.856112 7264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
I20260925 16:33:21.866338 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 7264
I20260925 16:33:21.866551 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 6716
Sep 25 16:33:21 dist-test-slave-65pf krb5kdc[6475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354001, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 25 16:33:21 dist-test-slave-65pf krb5kdc[6475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354001, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 25 16:33:21 dist-test-slave-65pf krb5kdc[6475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354001, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260925 16:33:21.873323 7242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:21.857904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34571 (local address 127.0.98.61:45951)
0925 16:33:21.858040 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:21.858044 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:21.859142 (+ 1098us) server_negotiation.cc:407] Connection header received
0925 16:33:21.860268 (+ 1126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:21.860272 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:21.860339 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:21.860394 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:21.861766 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.862601 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.864087 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.864396 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.864510 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:21.868683 (+ 4173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:21.868689 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:21.868693 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:21.868717 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:21.871014 (+ 2297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:21.871755 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:21.871761 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:21.871766 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:21.871845 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:21.872188 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:21.872194 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:21.872196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:21.872389 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:21.872497 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:21.873029 (+ 532us) server_negotiation.cc:299] Negotiation successful
0925 16:33:21.873151 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 16:33:21.873811 7491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:21.859449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35051 (local address 127.0.98.60:37381)
0925 16:33:21.859679 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:21.859682 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:21.859696 (+ 14us) server_negotiation.cc:407] Connection header received
0925 16:33:21.860257 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:21.860260 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:21.860319 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:21.860429 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:21.861753 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.862468 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.864108 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.864424 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.864519 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:21.868605 (+ 4086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:21.868608 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:21.868611 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:21.868642 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:21.871205 (+ 2563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:21.871714 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:21.871718 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:21.871720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:21.871813 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:21.872223 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:21.872229 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:21.872234 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:21.872493 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:21.872601 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:21.873376 (+ 775us) server_negotiation.cc:299] Negotiation successful
0925 16:33:21.873530 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":78,"threads_started":1}
I20260925 16:33:21.874699 7488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:21.858319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45951 (local address 127.0.98.1:34571)
0925 16:33:21.858934 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:21.859001 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:21.860035 (+ 1034us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:21.860609 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:21.860616 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:21.861133 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:21.861615 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:21.861623 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.862719 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.862726 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:21.863968 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:21.863979 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.864511 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.864515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:21.864612 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:21.865290 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:21.865309 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:21.868562 (+ 3253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:33:21.871146 (+ 2584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:21.871156 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:21.871176 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:21.871625 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:21.871959 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:21.871961 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:21.871963 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:21.872089 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:21.872501 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:33:21.872510 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:33:21.872774 (+ 264us) client_negotiation.cc:769] Sending connection context
0925 16:33:21.872970 (+ 196us) client_negotiation.cc:240] Negotiation successful
0925 16:33:21.873381 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"mutex_wait_us":254,"spinlock_wait_cycles":10240,"thread_start_us":700,"threads_started":1}
I20260925 16:33:21.874969 7490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:21.859340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37381 (local address 127.0.98.1:35051)
0925 16:33:21.859617 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:21.859629 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:21.860050 (+ 421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:21.860607 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:21.860614 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:21.861055 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:21.861604 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:21.861618 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.862648 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.862653 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:21.863968 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:21.863979 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.864642 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.864647 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:21.864762 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:21.865422 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:21.865434 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:21.868456 (+ 3022us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:33:21.871305 (+ 2849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:21.871309 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:21.871311 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:21.871610 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:21.871916 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:21.871922 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:21.871927 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:21.872100 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:21.872648 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:33:21.872654 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:33:21.872795 (+ 141us) client_negotiation.cc:769] Sending connection context
0925 16:33:21.872961 (+ 166us) client_negotiation.cc:240] Negotiation successful
0925 16:33:21.874056 (+ 1095us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":919,"spinlock_wait_cycles":6016,"thread_start_us":86,"threads_started":1}
I20260925 16:33:21.876410 7484 heartbeater.cc:347] Connected to a master server at 127.0.98.60:37381
I20260925 16:33:21.876701 7484 heartbeater.cc:464] Registering TS with master...
I20260925 16:33:21.877364 7484 heartbeater.cc:511] Master 127.0.98.60:37381 requested a full tablet report, sending...
I20260925 16:33:21.877343 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:43771
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:34575
--webserver_port=36409
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:34575,127.0.98.61:45951
--tserver_master_addrs=127.0.98.62:34575,127.0.98.61:45951,127.0.98.60:37381 with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 16:33:21.878713 7489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:21.858319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34575 (local address 127.0.98.1:48727)
0925 16:33:21.859324 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:21.859337 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:21.860041 (+ 704us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:21.860845 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:21.860851 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:21.861135 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:21.861615 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:21.861623 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.862739 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.862746 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:21.863741 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:21.863750 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.864306 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.864310 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:21.864505 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:21.865218 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:21.865234 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:21.868008 (+ 2774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:33:21.876655 (+ 8647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:21.876658 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:21.876659 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:21.876942 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:21.877270 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:21.877275 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:21.877277 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:21.877378 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:21.878134 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:33:21.878138 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:33:21.878247 (+ 109us) client_negotiation.cc:769] Sending connection context
0925 16:33:21.878376 (+ 129us) client_negotiation.cc:240] Negotiation successful
0925 16:33:21.878513 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":931,"mutex_wait_us":173,"spinlock_wait_cycles":22528,"thread_start_us":173,"threads_started":1}
I20260925 16:33:21.878854 6559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:21.857955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48727 (local address 127.0.98.62:34575)
0925 16:33:21.858081 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:21.858086 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:21.859414 (+ 1328us) server_negotiation.cc:407] Connection header received
0925 16:33:21.860366 (+ 952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:21.860372 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:21.860426 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:21.860496 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:21.861767 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.862600 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.863876 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:21.864161 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:21.864261 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:21.868180 (+ 3919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:21.868184 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:21.868186 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:21.868217 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:21.876552 (+ 8335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:21.877063 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:21.877065 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:21.877067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:21.877171 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:21.877835 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:21.877838 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:21.877839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:21.878015 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:21.878134 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:21.878463 (+ 329us) server_negotiation.cc:299] Negotiation successful
0925 16:33:21.878636 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 16:33:21.878866 7486 heartbeater.cc:347] Connected to a master server at 127.0.98.61:45951
I20260925 16:33:21.878945 7486 heartbeater.cc:464] Registering TS with master...
I20260925 16:33:21.879052 7305 ts_manager.cc:194] Registered new tserver with Master: 37e3504cf5f848cc912031815590b8a8 (127.0.98.1:43207)
I20260925 16:33:21.879127 7486 heartbeater.cc:511] Master 127.0.98.61:45951 requested a full tablet report, sending...
I20260925 16:33:21.879901 7485 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34575
I20260925 16:33:21.879980 7485 heartbeater.cc:464] Registering TS with master...
I20260925 16:33:21.880117 7485 heartbeater.cc:511] Master 127.0.98.62:34575 requested a full tablet report, sending...
I20260925 16:33:21.880216 7120 ts_manager.cc:194] Registered new tserver with Master: 37e3504cf5f848cc912031815590b8a8 (127.0.98.1:43207)
I20260925 16:33:21.880863 6520 ts_manager.cc:194] Re-registered known tserver with Master: 37e3504cf5f848cc912031815590b8a8 (127.0.98.1:43207)
I20260925 16:33:21.881419 6520 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48727
W20260925 16:33:21.995021 7492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:33:21.995271 7492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:33:21.995345 7492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 16:33:21.999097 6979 heartbeater.cc:503] Master 127.0.98.62:34575 was elected leader, sending a full tablet report...
W20260925 16:33:21.999743 7492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:33:21.999804 7492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:33:21.999912 7492 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 16:33:22.005424 7492 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:43771
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36409
--tserver_master_addrs=127.0.98.62:34575,127.0.98.61:45951,127.0.98.60:37381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7492
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:33:22.006453 7492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:33:22.007488 7492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:33:22.017354 7500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:22.017315 7497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:22.017524 7492 server_base.cc:1034] running on GCE node
W20260925 16:33:22.017315 7498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:22.017941 7492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:33:22.018546 7492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:33:22.019744 7492 hybrid_clock.cc:648] HybridClock initialized: now 1790354002019714 us; error 45 us; skew 500 ppm
Sep 25 16:33:22 dist-test-slave-65pf krb5kdc[6475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354002, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:33:22.022347 7492 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260925 16:33:22.023211 7492 webserver.cc:460] Webserver started at http://127.0.98.2:36409/ using document root <none> and password file <none>
I20260925 16:33:22.023751 7492 fs_manager.cc:359] Metadata directory not provided
I20260925 16:33:22.023856 7492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:33:22.027534 7492 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.001s
I20260925 16:33:22.029505 7507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:22.030783 7492 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:33:22.030901 7492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-1/wal
uuid: "0e11f3b8048a40d093ebd83dfdfe0b46"
format_stamp: "Formatted at 2026-09-25 16:33:20 on dist-test-slave-65pf"
server_key: "c48ed1e054ed5e68dd3571ebda78efa0"
server_key_iv: "3404f5623c02d64ac6dd96e16bc7f187"
server_key_version: "encryptionkey@0"
I20260925 16:33:22.031471 7492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:33:22.047478 7492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:33:22.050249 7492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:33:22.050457 7492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:33:22.051191 7492 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:33:22.052026 7492 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:33:22.052096 7492 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:22.052167 7492 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:33:22.052219 7492 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:22.061223 7492 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43771
I20260925 16:33:22.061247 7620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43771 every 8 connection(s)
I20260925 16:33:22.062173 7492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
I20260925 16:33:22.066097 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 7492
I20260925 16:33:22.066287 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 6850
Sep 25 16:33:22 dist-test-slave-65pf krb5kdc[6475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354002, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 25 16:33:22 dist-test-slave-65pf krb5kdc[6475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354002, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 25 16:33:22 dist-test-slave-65pf krb5kdc[6475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354002, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260925 16:33:22.077040 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:42375
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:34575
--webserver_port=43395
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:34575,127.0.98.61:45951
--tserver_master_addrs=127.0.98.62:34575,127.0.98.61:45951,127.0.98.60:37381 with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 16:33:22.080662 7491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:22.063844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60975 (local address 127.0.98.60:37381)
0925 16:33:22.063980 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:22.063984 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:22.064936 (+ 952us) server_negotiation.cc:407] Connection header received
0925 16:33:22.066396 (+ 1460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:22.066402 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:22.066473 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:22.066561 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:22.068650 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:22.069462 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:22.071514 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:22.071863 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:22.071937 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:22.076714 (+ 4777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:22.076719 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:22.076722 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:22.076757 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:22.078707 (+ 1950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:22.079176 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:22.079179 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:22.079181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:22.079282 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:22.079591 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:22.079595 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:22.079596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:22.079817 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:22.079896 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:22.080321 (+ 425us) server_negotiation.cc:299] Negotiation successful
0925 16:33:22.080486 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 16:33:22.080739 6559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:22.065438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56473 (local address 127.0.98.62:34575)
0925 16:33:22.065567 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:22.065573 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:22.065679 (+ 106us) server_negotiation.cc:407] Connection header received
0925 16:33:22.066389 (+ 710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:22.066395 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:22.066460 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:22.066516 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:22.068659 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:22.069568 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:22.071540 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:22.071981 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:22.072080 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:22.075721 (+ 3641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:22.075728 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:22.075735 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:22.075770 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:22.077942 (+ 2172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:22.078700 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:22.078706 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:22.078708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:22.078806 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:22.079240 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:22.079245 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:22.079248 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:22.079494 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:22.079609 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:22.080335 (+ 726us) server_negotiation.cc:299] Negotiation successful
0925 16:33:22.080480 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 16:33:22.080811 7242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:22.063847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39631 (local address 127.0.98.61:45951)
0925 16:33:22.063978 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:22.063983 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:22.065162 (+ 1179us) server_negotiation.cc:407] Connection header received
0925 16:33:22.066883 (+ 1721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:22.066887 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:22.066944 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:22.067003 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:22.068651 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:22.069466 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:22.071514 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:22.071858 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:22.071937 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:22.076019 (+ 4082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:22.076023 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:22.076026 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:22.076056 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:22.078794 (+ 2738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:22.079533 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:22.079537 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:22.079540 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:22.079620 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:22.079931 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:22.079935 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:22.079937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:22.080142 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:22.080215 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:22.080560 (+ 345us) server_negotiation.cc:299] Negotiation successful
0925 16:33:22.080671 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 16:33:22.082559 7625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:22.064263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37381 (local address 127.0.98.2:60975)
0925 16:33:22.064771 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:22.064821 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:22.066097 (+ 1276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:22.066958 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:22.066967 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:22.067624 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:22.068489 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:22.068499 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:22.069644 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:22.069648 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:22.071383 (+ 1735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:22.071389 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:22.072147 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:22.072151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:22.072453 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:22.073376 (+ 923us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:22.073418 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:22.076501 (+ 3083us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:33:22.078835 (+ 2334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:22.078840 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:22.078844 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:22.079063 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:22.079401 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:22.079404 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:22.079406 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:22.079491 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:22.079948 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:33:22.079950 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:33:22.080042 (+ 92us) client_negotiation.cc:769] Sending connection context
0925 16:33:22.081282 (+ 1240us) client_negotiation.cc:240] Negotiation successful
0925 16:33:22.081422 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":449,"spinlock_wait_cycles":18560,"thread_start_us":546,"threads_started":1}
I20260925 16:33:22.082561 7627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:22.065291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34575 (local address 127.0.98.2:56473)
0925 16:33:22.065560 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:22.065574 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:22.066143 (+ 569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:22.066955 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:22.066965 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:22.067666 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:22.068487 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:22.068501 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:22.069720 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:22.069726 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:22.071386 (+ 1660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:22.071400 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:22.072285 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:22.072292 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:22.072512 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:22.073544 (+ 1032us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:22.073560 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:22.075510 (+ 1950us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:33:22.078079 (+ 2569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:22.078088 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:22.078110 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:22.078512 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:22.078921 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:22.078926 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:22.078930 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:22.079126 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:22.079682 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:33:22.079691 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:33:22.080040 (+ 349us) client_negotiation.cc:769] Sending connection context
0925 16:33:22.080955 (+ 915us) client_negotiation.cc:240] Negotiation successful
0925 16:33:22.081510 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"mutex_wait_us":538,"spinlock_wait_cycles":12032,"thread_start_us":106,"threads_started":1}
I20260925 16:33:22.082868 7626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:22.064263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45951 (local address 127.0.98.2:39631)
0925 16:33:22.065072 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:22.065085 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:22.066143 (+ 1058us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:22.067179 (+ 1036us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:22.067181 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:22.067624 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:22.068478 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:22.068499 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:22.069644 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:22.069648 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:22.071368 (+ 1720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:22.071387 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:22.072147 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:22.072151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:22.072456 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:22.073376 (+ 920us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:22.073399 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:22.075836 (+ 2437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:33:22.079181 (+ 3345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:22.079183 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:22.079185 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:22.079423 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:22.079734 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:22.079737 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:22.079739 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:22.079812 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:22.080285 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:33:22.080288 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:33:22.080377 (+ 89us) client_negotiation.cc:769] Sending connection context
0925 16:33:22.081077 (+ 700us) client_negotiation.cc:240] Negotiation successful
0925 16:33:22.081375 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":731,"mutex_wait_us":564,"spinlock_wait_cycles":33536,"thread_start_us":163,"threads_started":1}
I20260925 16:33:22.084328 7621 heartbeater.cc:347] Connected to a master server at 127.0.98.60:37381
I20260925 16:33:22.084354 7623 heartbeater.cc:347] Connected to a master server at 127.0.98.61:45951
I20260925 16:33:22.084565 7621 heartbeater.cc:464] Registering TS with master...
I20260925 16:33:22.084568 7623 heartbeater.cc:464] Registering TS with master...
I20260925 16:33:22.084977 7622 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34575
I20260925 16:33:22.085047 7622 heartbeater.cc:464] Registering TS with master...
I20260925 16:33:22.085170 7622 heartbeater.cc:511] Master 127.0.98.62:34575 requested a full tablet report, sending...
I20260925 16:33:22.085145 7621 heartbeater.cc:511] Master 127.0.98.60:37381 requested a full tablet report, sending...
I20260925 16:33:22.085148 7623 heartbeater.cc:511] Master 127.0.98.61:45951 requested a full tablet report, sending...
I20260925 16:33:22.086282 6520 ts_manager.cc:194] Re-registered known tserver with Master: 0e11f3b8048a40d093ebd83dfdfe0b46 (127.0.98.2:43771)
I20260925 16:33:22.086556 7120 ts_manager.cc:194] Registered new tserver with Master: 0e11f3b8048a40d093ebd83dfdfe0b46 (127.0.98.2:43771)
I20260925 16:33:22.087132 6520 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56473
I20260925 16:33:22.087615 7305 ts_manager.cc:194] Registered new tserver with Master: 0e11f3b8048a40d093ebd83dfdfe0b46 (127.0.98.2:43771)
I20260925 16:33:22.183045 7629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:22.179106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37381 (local address 127.0.0.1:53548)
0925 16:33:22.179399 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:22.179413 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:22.179528 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:22.179836 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:22.179838 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:22.179851 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:33:22.179943 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:22.179947 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:22.180939 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:22.180941 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:22.181695 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:22.181703 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:22.182651 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:22.182656 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:22.182765 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:22.182786 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 16:33:22.182853 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 16:33:22.182913 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":75,"threads_started":1}
I20260925 16:33:22.183079 7491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:22.179208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53548 (local address 127.0.98.60:37381)
0925 16:33:22.179354 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:22.179359 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:22.179490 (+ 131us) server_negotiation.cc:407] Connection header received
0925 16:33:22.179628 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:22.179631 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:22.179696 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:22.179809 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:33:22.180036 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:22.180831 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:22.181819 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:22.182478 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:22.182630 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:22.182782 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:22.182868 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 16:33:22.182935 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 16:33:22.183688 7315 raft_consensus.cc:1203] T 00000000000000000000000000000000 P fffc9052c3ea4c7da2a350f0f2ca6d75 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260925 16:33:22.184094 7344 sys_catalog.cc:455] T 00000000000000000000000000000000 P fffc9052c3ea4c7da2a350f0f2ca6d75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6772881645114340bec15e5a7e09e6e5. Latest consensus state: current_term: 1 leader_uuid: "6772881645114340bec15e5a7e09e6e5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6772881645114340bec15e5a7e09e6e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34575 } } peers { permanent_uuid: "9fbd60470ceb46e2bb9f413b41a66ddb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45951 } attrs { promote: false } } peers { permanent_uuid: "fffc9052c3ea4c7da2a350f0f2ca6d75" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37381 } attrs { promote: true } } }
I20260925 16:33:22.184260 7344 sys_catalog.cc:458] T 00000000000000000000000000000000 P fffc9052c3ea4c7da2a350f0f2ca6d75 [sys.catalog]: This master's current role is: LEARNER
W20260925 16:33:22.196254 7628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:33:22.196506 7628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:33:22.196549 7628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:33:22.200627 7628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:33:22.200680 7628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:33:22.200739 7628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 16:33:22.205881 7628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:42375
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=43395
--tserver_master_addrs=127.0.98.62:34575,127.0.98.61:45951,127.0.98.60:37381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:33:22.206952 7628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:33:22.207923 7628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:33:22.217679 7635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:22.217769 7628 server_base.cc:1034] running on GCE node
W20260925 16:33:22.217671 7637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:22.217679 7634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:22.218403 7628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:33:22.219137 7628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:33:22.220293 7628 hybrid_clock.cc:648] HybridClock initialized: now 1790354002220266 us; error 40 us; skew 500 ppm
Sep 25 16:33:22 dist-test-slave-65pf krb5kdc[6475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354002, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:33:22.223133 7628 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260925 16:33:22.224179 7628 webserver.cc:460] Webserver started at http://127.0.98.3:43395/ using document root <none> and password file <none>
I20260925 16:33:22.224782 7628 fs_manager.cc:359] Metadata directory not provided
I20260925 16:33:22.224879 7628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:33:22.228813 7628 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 16:33:22.230829 7644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:22.232034 7628 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:33:22.232124 7628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-2/wal
uuid: "8d93ff6ea52e43f295aa3c9846e05303"
format_stamp: "Formatted at 2026-09-25 16:33:20 on dist-test-slave-65pf"
server_key: "4efd14369c81f051945e0e662c4363d2"
server_key_iv: "b2b69ed619d2cd6b973de6ad7b904f1e"
server_key_version: "encryptionkey@0"
I20260925 16:33:22.232623 7628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:33:22.244330 7628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:33:22.247639 7628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:33:22.247839 7628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:33:22.248425 7628 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:33:22.249253 7628 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:33:22.249298 7628 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:22.249360 7628 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:33:22.249392 7628 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:22.258864 7628 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42375
I20260925 16:33:22.258941 7757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42375 every 8 connection(s)
I20260925 16:33:22.259872 7628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
I20260925 16:33:22.264060 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 7628
Sep 25 16:33:22 dist-test-slave-65pf krb5kdc[6475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354002, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 25 16:33:22 dist-test-slave-65pf krb5kdc[6475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354002, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 25 16:33:22 dist-test-slave-65pf krb5kdc[6475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354002, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260925 16:33:22.278968 7491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:22.263164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57177 (local address 127.0.98.60:37381)
0925 16:33:22.263276 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:22.263281 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:22.263401 (+ 120us) server_negotiation.cc:407] Connection header received
0925 16:33:22.264002 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:22.264005 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:22.264064 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:22.264117 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:22.265887 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:22.266665 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:22.268757 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:22.269093 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:22.269191 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:22.273568 (+ 4377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:22.273572 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:22.273574 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:22.273606 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:22.275833 (+ 2227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:22.276447 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:22.276465 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:22.276469 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:22.276554 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:22.277006 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:22.277009 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:22.277011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:22.277294 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:22.277394 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:22.278638 (+ 1244us) server_negotiation.cc:299] Negotiation successful
0925 16:33:22.278772 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 16:33:22.280225 7762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:22.262055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45951 (local address 127.0.98.3:35651)
0925 16:33:22.262658 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:22.262724 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:22.263740 (+ 1016us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:22.264450 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:22.264461 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:22.265073 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:22.266962 (+ 1889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:22.266969 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:22.268027 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:22.268031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:22.268996 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:22.269007 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:22.269755 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:22.269759 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:22.269858 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:22.270654 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:22.270688 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:22.274188 (+ 3500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:33:22.277412 (+ 3224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:22.277417 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:22.277420 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:22.277671 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:22.278023 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:22.278026 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:22.278028 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:22.278119 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:22.278995 (+ 876us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:33:22.278998 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:33:22.279334 (+ 336us) client_negotiation.cc:769] Sending connection context
0925 16:33:22.279438 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 16:33:22.279580 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"mutex_wait_us":1689,"spinlock_wait_cycles":3840,"thread_start_us":701,"threads_started":1}
I20260925 16:33:22.280339 7771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:22.266806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34732 (local address 127.0.98.62:34575)
0925 16:33:22.268493 (+ 1687us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:22.268497 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:22.268510 (+ 13us) server_negotiation.cc:407] Connection header received
0925 16:33:22.268545 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:22.268548 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:22.268587 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:22.268665 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:22.269651 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:22.270399 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:22.273057 (+ 2658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:22.273407 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:22.273503 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:22.275434 (+ 1931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:22.275438 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:22.275441 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:22.275466 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:22.278050 (+ 2584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:22.278571 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:22.278574 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:22.278595 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:22.278681 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:22.278993 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:22.278997 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:22.278998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:22.279280 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:22.279408 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:22.280036 (+ 628us) server_negotiation.cc:299] Negotiation successful
0925 16:33:22.280154 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1620,"thread_start_us":78,"threads_started":1}
I20260925 16:33:22.280887 7242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:22.261628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35651 (local address 127.0.98.61:45951)
0925 16:33:22.261757 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:22.261761 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:22.262845 (+ 1084us) server_negotiation.cc:407] Connection header received
0925 16:33:22.264011 (+ 1166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:22.264016 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:22.264078 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:22.264139 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:22.267113 (+ 2974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:22.267866 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:22.269144 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:22.269516 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:22.269604 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:22.274309 (+ 4705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:22.274315 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:22.274319 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:22.274344 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:22.276785 (+ 2441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:22.277791 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:22.277796 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:22.277800 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:22.277888 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:22.278221 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:22.278224 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:22.278227 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:22.278453 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:22.278551 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:22.280636 (+ 2085us) server_negotiation.cc:299] Negotiation successful
0925 16:33:22.280742 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 16:33:22.281492 7769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:22.266198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41568 (local address 127.0.98.61:45951)
0925 16:33:22.270610 (+ 4412us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:22.270615 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:22.270627 (+ 12us) server_negotiation.cc:407] Connection header received
0925 16:33:22.270661 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:22.270663 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:22.270704 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:22.270788 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:22.271732 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:22.272574 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:22.275600 (+ 3026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:22.276003 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:22.276098 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:22.277639 (+ 1541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:22.277643 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:22.277645 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:22.277668 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:22.279393 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:22.279860 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:22.279863 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:22.279865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:22.279945 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:22.280249 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:22.280252 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:22.280254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:22.280468 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:22.280588 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:22.281208 (+ 620us) server_negotiation.cc:299] Negotiation successful
0925 16:33:22.281321 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3657,"thread_start_us":92,"threads_started":1}
I20260925 16:33:22.280234 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:22.263049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37381 (local address 127.0.98.3:57177)
0925 16:33:22.263317 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:22.263328 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:22.263765 (+ 437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:22.264449 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:22.264466 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:22.265071 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:22.265743 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:22.265758 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:22.266834 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:22.266836 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:22.268618 (+ 1782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:22.268628 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:22.269350 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:22.269354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:22.269727 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:22.270839 (+ 1112us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:22.270853 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:22.273380 (+ 2527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:33:22.275950 (+ 2570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:22.275956 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:22.275975 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:22.276324 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:22.276678 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:22.276681 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:22.276684 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:22.276872 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:22.277490 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:33:22.277497 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:33:22.277822 (+ 325us) client_negotiation.cc:769] Sending connection context
0925 16:33:22.278046 (+ 224us) client_negotiation.cc:240] Negotiation successful
0925 16:33:22.278427 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":383,"spinlock_wait_cycles":14208,"thread_start_us":78,"threads_started":1}
I20260925 16:33:22.282248 7760 heartbeater.cc:347] Connected to a master server at 127.0.98.61:45951
I20260925 16:33:22.282490 7760 heartbeater.cc:464] Registering TS with master...
I20260925 16:33:22.282780 7758 heartbeater.cc:347] Connected to a master server at 127.0.98.60:37381
I20260925 16:33:22.282848 7758 heartbeater.cc:464] Registering TS with master...
I20260925 16:33:22.283075 7760 heartbeater.cc:511] Master 127.0.98.61:45951 requested a full tablet report, sending...
I20260925 16:33:22.283075 7758 heartbeater.cc:511] Master 127.0.98.60:37381 requested a full tablet report, sending...
I20260925 16:33:22.284134 7120 ts_manager.cc:194] Registered new tserver with Master: 8d93ff6ea52e43f295aa3c9846e05303 (127.0.98.3:42375)
I20260925 16:33:22.284161 7305 ts_manager.cc:194] Registered new tserver with Master: 8d93ff6ea52e43f295aa3c9846e05303 (127.0.98.3:42375)
I20260925 16:33:22.284235 6559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:22.261617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54701 (local address 127.0.98.62:34575)
0925 16:33:22.261760 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:22.261764 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:22.263080 (+ 1316us) server_negotiation.cc:407] Connection header received
0925 16:33:22.264084 (+ 1004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:22.264089 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:22.264141 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:22.264218 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:22.265914 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:22.266722 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:22.268757 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:22.269090 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:22.269191 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:22.274562 (+ 5371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:22.274566 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:22.274569 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:22.274624 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:22.278887 (+ 4263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:22.281377 (+ 2490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:22.281382 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:22.281383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:22.281476 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:22.281883 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:22.281886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:22.281888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:22.282111 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:22.282195 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:22.283980 (+ 1785us) server_negotiation.cc:299] Negotiation successful
0925 16:33:22.284090 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 16:33:22.284677 7763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:22.262055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34575 (local address 127.0.98.3:54701)
0925 16:33:22.262999 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:22.263013 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:22.263767 (+ 754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:22.264449 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:22.264458 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:22.265071 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:22.265745 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:22.265761 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:22.266833 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:22.266836 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:22.268618 (+ 1782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:22.268628 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:22.269350 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:22.269354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:22.269725 (+ 371us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:22.270972 (+ 1247us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:22.270989 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:22.274388 (+ 3399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:33:22.281033 (+ 6645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:22.281036 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:22.281038 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:22.281288 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:22.281688 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:22.281691 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:22.281693 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:22.281777 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:22.283673 (+ 1896us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:33:22.283676 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:33:22.283783 (+ 107us) client_negotiation.cc:769] Sending connection context
0925 16:33:22.284340 (+ 557us) client_negotiation.cc:240] Negotiation successful
0925 16:33:22.284496 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":841,"mutex_wait_us":432,"spinlock_wait_cycles":37120,"thread_start_us":129,"threads_started":1}
I20260925 16:33:22.285393 7759 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34575
I20260925 16:33:22.285475 7759 heartbeater.cc:464] Registering TS with master...
I20260925 16:33:22.285632 7759 heartbeater.cc:511] Master 127.0.98.62:34575 requested a full tablet report, sending...
I20260925 16:33:22.286229 6519 ts_manager.cc:194] Re-registered known tserver with Master: 8d93ff6ea52e43f295aa3c9846e05303 (127.0.98.3:42375)
I20260925 16:33:22.283867 6520 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34732:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 16:33:22.287012 6519 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54701
W20260925 16:33:22.288167 6520 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:33:22.290552 7773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:22.267511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53558 (local address 127.0.98.60:37381)
0925 16:33:22.268033 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:22.268037 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:22.270379 (+ 2342us) server_negotiation.cc:407] Connection header received
0925 16:33:22.270419 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:22.270424 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:22.270489 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:22.270567 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:22.271516 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:22.272380 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:22.273650 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:22.273998 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:22.274096 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:22.285870 (+ 11774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:22.285873 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:22.285876 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:22.285897 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:22.288983 (+ 3086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:22.289390 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:22.289394 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:22.289395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:22.289484 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:22.289737 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:22.289740 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:22.289742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:22.289967 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:22.290097 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:22.290307 (+ 210us) server_negotiation.cc:299] Negotiation successful
0925 16:33:22.290401 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":85,"threads_started":1}
I20260925 16:33:22.291070 7271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6772881645114340bec15e5a7e09e6e5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6772881645114340bec15e5a7e09e6e5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6772881645114340bec15e5a7e09e6e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34575 } } peers { permanent_uuid: "9fbd60470ceb46e2bb9f413b41a66ddb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45951 } attrs { promote: false } } peers { permanent_uuid: "fffc9052c3ea4c7da2a350f0f2ca6d75" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37381 } attrs { promote: true } } }
I20260925 16:33:22.291188 7271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6772881645114340bec15e5a7e09e6e5 [sys.catalog]: This master's current role is: LEADER
I20260925 16:33:22.294813 7777 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 6772881645114340bec15e5a7e09e6e5: attempting to promote NON_VOTER fffc9052c3ea4c7da2a350f0f2ca6d75 to VOTER
I20260925 16:33:22.295217 7777 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6772881645114340bec15e5a7e09e6e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6772881645114340bec15e5a7e09e6e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34575 } } peers { permanent_uuid: "9fbd60470ceb46e2bb9f413b41a66ddb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45951 } attrs { promote: false } } peers { permanent_uuid: "fffc9052c3ea4c7da2a350f0f2ca6d75" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37381 } attrs { promote: false } }
I20260925 16:33:22.296214 7130 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb [term 1 FOLLOWER]: Refusing update from remote peer 6772881645114340bec15e5a7e09e6e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260925 16:33:22.296630 7776 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6772881645114340bec15e5a7e09e6e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fbd60470ceb46e2bb9f413b41a66ddb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45951 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260925 16:33:22.296960 7314 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fffc9052c3ea4c7da2a350f0f2ca6d75 [term 1 LEARNER]: Refusing update from remote peer 6772881645114340bec15e5a7e09e6e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260925 16:33:22.297338 7776 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6772881645114340bec15e5a7e09e6e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fffc9052c3ea4c7da2a350f0f2ca6d75" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260925 16:33:22.298200 7271 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6772881645114340bec15e5a7e09e6e5 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, fffc9052c3ea4c7da2a350f0f2ca6d75 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6772881645114340bec15e5a7e09e6e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34575 } } peers { permanent_uuid: "9fbd60470ceb46e2bb9f413b41a66ddb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45951 } attrs { promote: false } } peers { permanent_uuid: "fffc9052c3ea4c7da2a350f0f2ca6d75" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37381 } attrs { promote: false } } }
I20260925 16:33:22.298645 7130 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, fffc9052c3ea4c7da2a350f0f2ca6d75 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6772881645114340bec15e5a7e09e6e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34575 } } peers { permanent_uuid: "9fbd60470ceb46e2bb9f413b41a66ddb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45951 } attrs { promote: false } } peers { permanent_uuid: "fffc9052c3ea4c7da2a350f0f2ca6d75" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37381 } attrs { promote: false } } }
I20260925 16:33:22.299719 7776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6772881645114340bec15e5a7e09e6e5 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6772881645114340bec15e5a7e09e6e5" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6772881645114340bec15e5a7e09e6e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34575 } } peers { permanent_uuid: "9fbd60470ceb46e2bb9f413b41a66ddb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45951 } attrs { promote: false } } peers { permanent_uuid: "fffc9052c3ea4c7da2a350f0f2ca6d75" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37381 } attrs { promote: false } } }
I20260925 16:33:22.299832 7776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6772881645114340bec15e5a7e09e6e5 [sys.catalog]: This master's current role is: LEADER
I20260925 16:33:22.299755 7271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6772881645114340bec15e5a7e09e6e5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6772881645114340bec15e5a7e09e6e5" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6772881645114340bec15e5a7e09e6e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34575 } } peers { permanent_uuid: "9fbd60470ceb46e2bb9f413b41a66ddb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45951 } attrs { promote: false } } peers { permanent_uuid: "fffc9052c3ea4c7da2a350f0f2ca6d75" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37381 } attrs { promote: false } } }
I20260925 16:33:22.299880 7271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6772881645114340bec15e5a7e09e6e5 [sys.catalog]: This master's current role is: LEADER
I20260925 16:33:22.299937 7776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6772881645114340bec15e5a7e09e6e5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6772881645114340bec15e5a7e09e6e5" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6772881645114340bec15e5a7e09e6e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34575 } } peers { permanent_uuid: "9fbd60470ceb46e2bb9f413b41a66ddb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45951 } attrs { promote: false } } peers { permanent_uuid: "fffc9052c3ea4c7da2a350f0f2ca6d75" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37381 } attrs { promote: false } } }
I20260925 16:33:22.300000 7776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6772881645114340bec15e5a7e09e6e5 [sys.catalog]: This master's current role is: LEADER
I20260925 16:33:22.299774 7314 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fffc9052c3ea4c7da2a350f0f2ca6d75 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, fffc9052c3ea4c7da2a350f0f2ca6d75 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6772881645114340bec15e5a7e09e6e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34575 } } peers { permanent_uuid: "9fbd60470ceb46e2bb9f413b41a66ddb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45951 } attrs { promote: false } } peers { permanent_uuid: "fffc9052c3ea4c7da2a350f0f2ca6d75" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37381 } attrs { promote: false } } }
I20260925 16:33:22.300561 7785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6772881645114340bec15e5a7e09e6e5" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6772881645114340bec15e5a7e09e6e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34575 } } peers { permanent_uuid: "9fbd60470ceb46e2bb9f413b41a66ddb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45951 } attrs { promote: false } } peers { permanent_uuid: "fffc9052c3ea4c7da2a350f0f2ca6d75" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37381 } attrs { promote: false } } }
I20260925 16:33:22.300722 7785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:33:22.301558 7344 sys_catalog.cc:455] T 00000000000000000000000000000000 P fffc9052c3ea4c7da2a350f0f2ca6d75 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6772881645114340bec15e5a7e09e6e5" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6772881645114340bec15e5a7e09e6e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34575 } } peers { permanent_uuid: "9fbd60470ceb46e2bb9f413b41a66ddb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45951 } attrs { promote: false } } peers { permanent_uuid: "fffc9052c3ea4c7da2a350f0f2ca6d75" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37381 } attrs { promote: false } } }
I20260925 16:33:22.301656 7344 sys_catalog.cc:458] T 00000000000000000000000000000000 P fffc9052c3ea4c7da2a350f0f2ca6d75 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:33:22.309015 7629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:22.304175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42375 (local address 127.0.0.1:57112)
0925 16:33:22.304322 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:22.304338 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:22.304422 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:22.305185 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:22.305188 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:22.305207 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:33:22.305308 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:22.305315 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:22.306854 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:22.306861 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:22.307621 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:22.307628 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:22.308652 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:22.308656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:22.308747 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:22.308774 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 16:33:22.308818 (+ 44us) client_negotiation.cc:240] Negotiation successful
0925 16:33:22.308887 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20260925 16:33:22.309741 7786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:22.304347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57112 (local address 127.0.98.3:42375)
0925 16:33:22.304656 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:22.304668 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:22.304694 (+ 26us) server_negotiation.cc:407] Connection header received
0925 16:33:22.304777 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:22.304783 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:22.305019 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:22.305483 (+ 464us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:33:22.305649 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:22.306756 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:22.307744 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:22.308516 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:22.309285 (+ 769us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:22.309435 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:22.309498 (+ 63us) server_negotiation.cc:299] Negotiation successful
0925 16:33:22.309596 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":132,"threads_started":1}
I20260925 16:33:22.310638 7787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:22.304750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43207 (local address 127.0.0.1:38126)
0925 16:33:22.305330 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:22.305343 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:22.305429 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:22.306275 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:22.306278 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:22.306295 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:33:22.306373 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:22.306378 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:22.308159 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:22.308165 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:22.308988 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:22.308997 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:22.310177 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:22.310181 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:22.310282 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:22.310303 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 16:33:22.310373 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 16:33:22.310439 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":97,"threads_started":1}
I20260925 16:33:22.310650 7790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:22.305333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38126 (local address 127.0.98.1:43207)
0925 16:33:22.305693 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:22.305703 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:22.305725 (+ 22us) server_negotiation.cc:407] Connection header received
0925 16:33:22.305797 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:22.305801 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:22.306000 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:22.306144 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:33:22.306482 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:22.308043 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:22.309130 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:22.310044 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:22.310178 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:22.310312 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:22.310399 (+ 87us) server_negotiation.cc:299] Negotiation successful
0925 16:33:22.310493 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":133,"threads_started":1}
I20260925 16:33:22.312446 7789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:22.304750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43771 (local address 127.0.0.1:45946)
0925 16:33:22.306607 (+ 1857us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:22.306622 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:22.307177 (+ 555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:22.307827 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:22.307830 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:22.307845 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:33:22.307922 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:22.307927 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:22.309214 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:22.309220 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:22.311125 (+ 1905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:22.311135 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:22.312069 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:22.312072 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:22.312157 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:22.312178 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 16:33:22.312257 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 16:33:22.312323 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1780,"mutex_wait_us":319,"thread_start_us":72,"threads_started":1}
I20260925 16:33:22.312502 7788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:22.304979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45946 (local address 127.0.98.2:43771)
0925 16:33:22.306439 (+ 1460us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:22.306462 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:22.307276 (+ 814us) server_negotiation.cc:407] Connection header received
0925 16:33:22.307353 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:22.307358 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:22.307547 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:22.307681 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:33:22.308034 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:22.309085 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:22.311235 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:22.311932 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:22.312069 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:22.312200 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:22.312284 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 16:33:22.312365 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1355,"thread_start_us":108,"threads_started":1}
I20260925 16:33:22.312678 7692 tablet_service.cc:1467] Processing CreateTablet for tablet 21362dbfeedd4a6c908232ce6326aac6 (DEFAULT_TABLE table=test-table [id=280ad007d303440eada05cb25c91db5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:33:22.313921 7692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21362dbfeedd4a6c908232ce6326aac6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:33:22.313601 7418 tablet_service.cc:1467] Processing CreateTablet for tablet 21362dbfeedd4a6c908232ce6326aac6 (DEFAULT_TABLE table=test-table [id=280ad007d303440eada05cb25c91db5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:33:22.314630 7418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21362dbfeedd4a6c908232ce6326aac6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:33:22.315186 7555 tablet_service.cc:1467] Processing CreateTablet for tablet 21362dbfeedd4a6c908232ce6326aac6 (DEFAULT_TABLE table=test-table [id=280ad007d303440eada05cb25c91db5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:33:22.315948 7555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21362dbfeedd4a6c908232ce6326aac6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:33:22.322252 7791 tablet_bootstrap.cc:492] T 21362dbfeedd4a6c908232ce6326aac6 P 0e11f3b8048a40d093ebd83dfdfe0b46: Bootstrap starting.
I20260925 16:33:22.323443 7792 tablet_bootstrap.cc:492] T 21362dbfeedd4a6c908232ce6326aac6 P 37e3504cf5f848cc912031815590b8a8: Bootstrap starting.
I20260925 16:33:22.324522 7793 tablet_bootstrap.cc:492] T 21362dbfeedd4a6c908232ce6326aac6 P 8d93ff6ea52e43f295aa3c9846e05303: Bootstrap starting.
I20260925 16:33:22.324724 7791 tablet_bootstrap.cc:654] T 21362dbfeedd4a6c908232ce6326aac6 P 0e11f3b8048a40d093ebd83dfdfe0b46: Neither blocks nor log segments found. Creating new log.
I20260925 16:33:22.325541 7791 log.cc:824] T 21362dbfeedd4a6c908232ce6326aac6 P 0e11f3b8048a40d093ebd83dfdfe0b46: Log is configured to *not* fsync() on all Append() calls
I20260925 16:33:22.326356 7793 tablet_bootstrap.cc:654] T 21362dbfeedd4a6c908232ce6326aac6 P 8d93ff6ea52e43f295aa3c9846e05303: Neither blocks nor log segments found. Creating new log.
I20260925 16:33:22.327244 7793 log.cc:824] T 21362dbfeedd4a6c908232ce6326aac6 P 8d93ff6ea52e43f295aa3c9846e05303: Log is configured to *not* fsync() on all Append() calls
I20260925 16:33:22.327526 7791 tablet_bootstrap.cc:492] T 21362dbfeedd4a6c908232ce6326aac6 P 0e11f3b8048a40d093ebd83dfdfe0b46: No bootstrap required, opened a new log
I20260925 16:33:22.327725 7791 ts_tablet_manager.cc:1397] T 21362dbfeedd4a6c908232ce6326aac6 P 0e11f3b8048a40d093ebd83dfdfe0b46: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 16:33:22.327901 7792 tablet_bootstrap.cc:654] T 21362dbfeedd4a6c908232ce6326aac6 P 37e3504cf5f848cc912031815590b8a8: Neither blocks nor log segments found. Creating new log.
I20260925 16:33:22.328457 7792 log.cc:824] T 21362dbfeedd4a6c908232ce6326aac6 P 37e3504cf5f848cc912031815590b8a8: Log is configured to *not* fsync() on all Append() calls
I20260925 16:33:22.330016 7792 tablet_bootstrap.cc:492] T 21362dbfeedd4a6c908232ce6326aac6 P 37e3504cf5f848cc912031815590b8a8: No bootstrap required, opened a new log
I20260925 16:33:22.330158 7792 ts_tablet_manager.cc:1397] T 21362dbfeedd4a6c908232ce6326aac6 P 37e3504cf5f848cc912031815590b8a8: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260925 16:33:22.332196 7791 raft_consensus.cc:348] T 21362dbfeedd4a6c908232ce6326aac6 P 0e11f3b8048a40d093ebd83dfdfe0b46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d93ff6ea52e43f295aa3c9846e05303" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42375 } } peers { permanent_uuid: "0e11f3b8048a40d093ebd83dfdfe0b46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43771 } } peers { permanent_uuid: "37e3504cf5f848cc912031815590b8a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43207 } }
I20260925 16:33:22.332432 7791 raft_consensus.cc:374] T 21362dbfeedd4a6c908232ce6326aac6 P 0e11f3b8048a40d093ebd83dfdfe0b46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:33:22.332500 7791 raft_consensus.cc:729] T 21362dbfeedd4a6c908232ce6326aac6 P 0e11f3b8048a40d093ebd83dfdfe0b46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e11f3b8048a40d093ebd83dfdfe0b46, State: Initialized, Role: FOLLOWER
I20260925 16:33:22.332901 7791 consensus_queue.cc:260] T 21362dbfeedd4a6c908232ce6326aac6 P 0e11f3b8048a40d093ebd83dfdfe0b46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d93ff6ea52e43f295aa3c9846e05303" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42375 } } peers { permanent_uuid: "0e11f3b8048a40d093ebd83dfdfe0b46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43771 } } peers { permanent_uuid: "37e3504cf5f848cc912031815590b8a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43207 } }
I20260925 16:33:22.333560 7622 heartbeater.cc:503] Master 127.0.98.62:34575 was elected leader, sending a full tablet report...
I20260925 16:33:22.333895 7791 ts_tablet_manager.cc:1428] T 21362dbfeedd4a6c908232ce6326aac6 P 0e11f3b8048a40d093ebd83dfdfe0b46: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 16:33:22.334035 7792 raft_consensus.cc:348] T 21362dbfeedd4a6c908232ce6326aac6 P 37e3504cf5f848cc912031815590b8a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d93ff6ea52e43f295aa3c9846e05303" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42375 } } peers { permanent_uuid: "0e11f3b8048a40d093ebd83dfdfe0b46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43771 } } peers { permanent_uuid: "37e3504cf5f848cc912031815590b8a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43207 } }
I20260925 16:33:22.334267 7792 raft_consensus.cc:374] T 21362dbfeedd4a6c908232ce6326aac6 P 37e3504cf5f848cc912031815590b8a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:33:22.334367 7792 raft_consensus.cc:729] T 21362dbfeedd4a6c908232ce6326aac6 P 37e3504cf5f848cc912031815590b8a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37e3504cf5f848cc912031815590b8a8, State: Initialized, Role: FOLLOWER
I20260925 16:33:22.334807 7792 consensus_queue.cc:260] T 21362dbfeedd4a6c908232ce6326aac6 P 37e3504cf5f848cc912031815590b8a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d93ff6ea52e43f295aa3c9846e05303" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42375 } } peers { permanent_uuid: "0e11f3b8048a40d093ebd83dfdfe0b46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43771 } } peers { permanent_uuid: "37e3504cf5f848cc912031815590b8a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43207 } }
I20260925 16:33:22.335896 7792 ts_tablet_manager.cc:1428] T 21362dbfeedd4a6c908232ce6326aac6 P 37e3504cf5f848cc912031815590b8a8: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 16:33:22.336267 7793 tablet_bootstrap.cc:492] T 21362dbfeedd4a6c908232ce6326aac6 P 8d93ff6ea52e43f295aa3c9846e05303: No bootstrap required, opened a new log
I20260925 16:33:22.336464 7793 ts_tablet_manager.cc:1397] T 21362dbfeedd4a6c908232ce6326aac6 P 8d93ff6ea52e43f295aa3c9846e05303: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.003s
I20260925 16:33:22.337100 7485 heartbeater.cc:503] Master 127.0.98.62:34575 was elected leader, sending a full tablet report...
I20260925 16:33:22.340927 7793 raft_consensus.cc:348] T 21362dbfeedd4a6c908232ce6326aac6 P 8d93ff6ea52e43f295aa3c9846e05303 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d93ff6ea52e43f295aa3c9846e05303" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42375 } } peers { permanent_uuid: "0e11f3b8048a40d093ebd83dfdfe0b46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43771 } } peers { permanent_uuid: "37e3504cf5f848cc912031815590b8a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43207 } }
I20260925 16:33:22.341228 7793 raft_consensus.cc:374] T 21362dbfeedd4a6c908232ce6326aac6 P 8d93ff6ea52e43f295aa3c9846e05303 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:33:22.341288 7793 raft_consensus.cc:729] T 21362dbfeedd4a6c908232ce6326aac6 P 8d93ff6ea52e43f295aa3c9846e05303 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d93ff6ea52e43f295aa3c9846e05303, State: Initialized, Role: FOLLOWER
I20260925 16:33:22.341727 7793 consensus_queue.cc:260] T 21362dbfeedd4a6c908232ce6326aac6 P 8d93ff6ea52e43f295aa3c9846e05303 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d93ff6ea52e43f295aa3c9846e05303" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42375 } } peers { permanent_uuid: "0e11f3b8048a40d093ebd83dfdfe0b46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43771 } } peers { permanent_uuid: "37e3504cf5f848cc912031815590b8a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43207 } }
I20260925 16:33:22.342550 7759 heartbeater.cc:503] Master 127.0.98.62:34575 was elected leader, sending a full tablet report...
I20260925 16:33:22.342785 7793 ts_tablet_manager.cc:1428] T 21362dbfeedd4a6c908232ce6326aac6 P 8d93ff6ea52e43f295aa3c9846e05303: Time spent starting tablet: real 0.006s user 0.006s sys 0.002s
I20260925 16:33:22.352710 7797 raft_consensus.cc:482] T 21362dbfeedd4a6c908232ce6326aac6 P 0e11f3b8048a40d093ebd83dfdfe0b46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:33:22.352887 7797 raft_consensus.cc:504] T 21362dbfeedd4a6c908232ce6326aac6 P 0e11f3b8048a40d093ebd83dfdfe0b46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d93ff6ea52e43f295aa3c9846e05303" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42375 } } peers { permanent_uuid: "0e11f3b8048a40d093ebd83dfdfe0b46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43771 } } peers { permanent_uuid: "37e3504cf5f848cc912031815590b8a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43207 } }
I20260925 16:33:22.353976 7797 leader_election.cc:288] T 21362dbfeedd4a6c908232ce6326aac6 P 0e11f3b8048a40d093ebd83dfdfe0b46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d93ff6ea52e43f295aa3c9846e05303 (127.0.98.3:42375), 37e3504cf5f848cc912031815590b8a8 (127.0.98.1:43207)
W20260925 16:33:22.357592 7487 tablet.cc:2369] T 21362dbfeedd4a6c908232ce6326aac6 P 37e3504cf5f848cc912031815590b8a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:33:22.359083 7786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:22.354321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51605 (local address 127.0.98.3:42375)
0925 16:33:22.354437 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:22.354444 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:22.354458 (+ 14us) server_negotiation.cc:407] Connection header received
0925 16:33:22.354504 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:22.354509 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:22.354553 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:22.354656 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:33:22.355003 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:22.355864 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:22.357607 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:22.358177 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:22.358286 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:22.358346 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:22.358834 (+ 488us) server_negotiation.cc:299] Negotiation successful
0925 16:33:22.358907 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 16:33:22.359105 7627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:22.354132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42375 (local address 127.0.98.2:51605)
0925 16:33:22.354257 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:22.354273 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:22.354355 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:22.354668 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:22.354671 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:22.354689 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:33:22.354860 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:22.354867 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:22.355983 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:22.355986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:22.357452 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:22.357471 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:22.358314 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:22.358318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:22.358647 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:22.358682 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 16:33:22.358820 (+ 138us) client_negotiation.cc:240] Negotiation successful
0925 16:33:22.358919 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":89}
I20260925 16:33:22.360011 7712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21362dbfeedd4a6c908232ce6326aac6" candidate_uuid: "0e11f3b8048a40d093ebd83dfdfe0b46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d93ff6ea52e43f295aa3c9846e05303" is_pre_election: true
I20260925 16:33:22.360306 7712 raft_consensus.cc:2454] T 21362dbfeedd4a6c908232ce6326aac6 P 8d93ff6ea52e43f295aa3c9846e05303 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e11f3b8048a40d093ebd83dfdfe0b46 in term 0.
I20260925 16:33:22.360778 7510 leader_election.cc:302] T 21362dbfeedd4a6c908232ce6326aac6 P 0e11f3b8048a40d093ebd83dfdfe0b46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e11f3b8048a40d093ebd83dfdfe0b46, 8d93ff6ea52e43f295aa3c9846e05303; no voters:
I20260925 16:33:22.361052 7797 raft_consensus.cc:2781] T 21362dbfeedd4a6c908232ce6326aac6 P 0e11f3b8048a40d093ebd83dfdfe0b46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:33:22.361140 7797 raft_consensus.cc:482] T 21362dbfeedd4a6c908232ce6326aac6 P 0e11f3b8048a40d093ebd83dfdfe0b46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:33:22.361196 7797 raft_consensus.cc:3037] T 21362dbfeedd4a6c908232ce6326aac6 P 0e11f3b8048a40d093ebd83dfdfe0b46 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:33:22.362012 7790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:22.354380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33727 (local address 127.0.98.1:43207)
0925 16:33:22.354602 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:22.354608 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:22.357785 (+ 3177us) server_negotiation.cc:407] Connection header received
0925 16:33:22.358025 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:22.358029 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:22.358183 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:22.358247 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:33:22.358727 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:22.359444 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:22.360943 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:22.361470 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:22.361590 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:22.361646 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:22.361836 (+ 190us) server_negotiation.cc:299] Negotiation successful
0925 16:33:22.361904 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260925 16:33:22.362012 7626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:22.354718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43207 (local address 127.0.98.2:33727)
0925 16:33:22.357648 (+ 2930us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:22.357687 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:22.357868 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:22.358368 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:22.358371 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:22.358387 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:33:22.358560 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:22.358567 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:22.359555 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:22.359559 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:22.360808 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:22.360817 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:22.361590 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:22.361594 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:22.361694 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:22.361714 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 16:33:22.361820 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 16:33:22.361891 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2816,"mutex_wait_us":52}
I20260925 16:33:22.362322 7797 raft_consensus.cc:504] T 21362dbfeedd4a6c908232ce6326aac6 P 0e11f3b8048a40d093ebd83dfdfe0b46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d93ff6ea52e43f295aa3c9846e05303" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42375 } } peers { permanent_uuid: "0e11f3b8048a40d093ebd83dfdfe0b46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43771 } } peers { permanent_uuid: "37e3504cf5f848cc912031815590b8a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43207 } }
I20260925 16:33:22.362886 7438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21362dbfeedd4a6c908232ce6326aac6" candidate_uuid: "0e11f3b8048a40d093ebd83dfdfe0b46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37e3504cf5f848cc912031815590b8a8" is_pre_election: true
I20260925 16:33:22.363157 7438 raft_consensus.cc:2454] T 21362dbfeedd4a6c908232ce6326aac6 P 37e3504cf5f848cc912031815590b8a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e11f3b8048a40d093ebd83dfdfe0b46 in term 0.
I20260925 16:33:22.363186 7797 leader_election.cc:288] T 21362dbfeedd4a6c908232ce6326aac6 P 0e11f3b8048a40d093ebd83dfdfe0b46 [CANDIDATE]: Term 1 election: Requested vote from peers 8d93ff6ea52e43f295aa3c9846e05303 (127.0.98.3:42375), 37e3504cf5f848cc912031815590b8a8 (127.0.98.1:43207)
I20260925 16:33:22.363461 7712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21362dbfeedd4a6c908232ce6326aac6" candidate_uuid: "0e11f3b8048a40d093ebd83dfdfe0b46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d93ff6ea52e43f295aa3c9846e05303"
I20260925 16:33:22.363513 7438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21362dbfeedd4a6c908232ce6326aac6" candidate_uuid: "0e11f3b8048a40d093ebd83dfdfe0b46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37e3504cf5f848cc912031815590b8a8"
I20260925 16:33:22.363649 7438 raft_consensus.cc:3037] T 21362dbfeedd4a6c908232ce6326aac6 P 37e3504cf5f848cc912031815590b8a8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:33:22.363605 7712 raft_consensus.cc:3037] T 21362dbfeedd4a6c908232ce6326aac6 P 8d93ff6ea52e43f295aa3c9846e05303 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:33:22.364773 7438 raft_consensus.cc:2454] T 21362dbfeedd4a6c908232ce6326aac6 P 37e3504cf5f848cc912031815590b8a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e11f3b8048a40d093ebd83dfdfe0b46 in term 1.
I20260925 16:33:22.364851 7712 raft_consensus.cc:2454] T 21362dbfeedd4a6c908232ce6326aac6 P 8d93ff6ea52e43f295aa3c9846e05303 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e11f3b8048a40d093ebd83dfdfe0b46 in term 1.
I20260925 16:33:22.365028 7511 leader_election.cc:302] T 21362dbfeedd4a6c908232ce6326aac6 P 0e11f3b8048a40d093ebd83dfdfe0b46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e11f3b8048a40d093ebd83dfdfe0b46, 37e3504cf5f848cc912031815590b8a8; no voters:
I20260925 16:33:22.365187 7797 raft_consensus.cc:2781] T 21362dbfeedd4a6c908232ce6326aac6 P 0e11f3b8048a40d093ebd83dfdfe0b46 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:33:22.365490 7797 raft_consensus.cc:686] T 21362dbfeedd4a6c908232ce6326aac6 P 0e11f3b8048a40d093ebd83dfdfe0b46 [term 1 LEADER]: Becoming Leader. State: Replica: 0e11f3b8048a40d093ebd83dfdfe0b46, State: Running, Role: LEADER
I20260925 16:33:22.365808 7797 consensus_queue.cc:237] T 21362dbfeedd4a6c908232ce6326aac6 P 0e11f3b8048a40d093ebd83dfdfe0b46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d93ff6ea52e43f295aa3c9846e05303" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42375 } } peers { permanent_uuid: "0e11f3b8048a40d093ebd83dfdfe0b46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43771 } } peers { permanent_uuid: "37e3504cf5f848cc912031815590b8a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43207 } }
I20260925 16:33:22.370034 6519 catalog_manager.cc:5495] T 21362dbfeedd4a6c908232ce6326aac6 P 0e11f3b8048a40d093ebd83dfdfe0b46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e11f3b8048a40d093ebd83dfdfe0b46 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "0e11f3b8048a40d093ebd83dfdfe0b46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d93ff6ea52e43f295aa3c9846e05303" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e11f3b8048a40d093ebd83dfdfe0b46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37e3504cf5f848cc912031815590b8a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43207 } health_report { overall_health: UNKNOWN } } }
I20260925 16:33:22.385705 7788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:22.382659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45952 (local address 127.0.98.2:43771)
0925 16:33:22.382790 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:22.382793 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:22.382824 (+ 31us) server_negotiation.cc:407] Connection header received
0925 16:33:22.382986 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:22.382988 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:22.383041 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:22.383124 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:33:22.383302 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:22.383790 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:22.384554 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:22.384761 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:22.384831 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:22.385050 (+ 219us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:33:22.385418 (+ 368us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:33:22.385502 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:22.385564 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 16:33:22.385608 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 16:33:22.391767 7438 raft_consensus.cc:1261] T 21362dbfeedd4a6c908232ce6326aac6 P 37e3504cf5f848cc912031815590b8a8 [term 1 FOLLOWER]: Refusing update from remote peer 0e11f3b8048a40d093ebd83dfdfe0b46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:33:22.392247 7712 raft_consensus.cc:1261] T 21362dbfeedd4a6c908232ce6326aac6 P 8d93ff6ea52e43f295aa3c9846e05303 [term 1 FOLLOWER]: Refusing update from remote peer 0e11f3b8048a40d093ebd83dfdfe0b46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:33:22.392472 7797 consensus_queue.cc:1035] T 21362dbfeedd4a6c908232ce6326aac6 P 0e11f3b8048a40d093ebd83dfdfe0b46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37e3504cf5f848cc912031815590b8a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:33:22.392921 7800 consensus_queue.cc:1035] T 21362dbfeedd4a6c908232ce6326aac6 P 0e11f3b8048a40d093ebd83dfdfe0b46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d93ff6ea52e43f295aa3c9846e05303" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:33:22.404660 7805 mvcc.cc:205] Tried to move back new op lower bound from 7333289993788850176 to 7333289993693372416. Current Snapshot: MvccSnapshot[applied={T|T < 7333289993788850176}]
I20260925 16:33:22.411146 6519 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34732:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:33:22.411281 6519 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34732:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:33:22.416530 6519 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6772881645114340bec15e5a7e09e6e5: Sending DeleteTablet for 3 replicas of tablet 21362dbfeedd4a6c908232ce6326aac6
I20260925 16:33:22.417294 7418 tablet_service.cc:1514] Processing DeleteTablet for tablet 21362dbfeedd4a6c908232ce6326aac6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:33:22 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38126
I20260925 16:33:22.417371 7692 tablet_service.cc:1514] Processing DeleteTablet for tablet 21362dbfeedd4a6c908232ce6326aac6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:33:22 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57112
I20260925 16:33:22.417789 7813 tablet_replica.cc:330] T 21362dbfeedd4a6c908232ce6326aac6 P 37e3504cf5f848cc912031815590b8a8: stopping tablet replica
I20260925 16:33:22.417805 7555 tablet_service.cc:1514] Processing DeleteTablet for tablet 21362dbfeedd4a6c908232ce6326aac6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:33:22 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45946
I20260925 16:33:22.418212 7813 raft_consensus.cc:2229] T 21362dbfeedd4a6c908232ce6326aac6 P 37e3504cf5f848cc912031815590b8a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:33:22.418262 7814 tablet_replica.cc:330] T 21362dbfeedd4a6c908232ce6326aac6 P 8d93ff6ea52e43f295aa3c9846e05303: stopping tablet replica
I20260925 16:33:22.418560 7814 raft_consensus.cc:2229] T 21362dbfeedd4a6c908232ce6326aac6 P 8d93ff6ea52e43f295aa3c9846e05303 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:33:22.418627 7813 raft_consensus.cc:2258] T 21362dbfeedd4a6c908232ce6326aac6 P 37e3504cf5f848cc912031815590b8a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:33:22.418902 7814 raft_consensus.cc:2258] T 21362dbfeedd4a6c908232ce6326aac6 P 8d93ff6ea52e43f295aa3c9846e05303 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:33:22.420166 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 6488
I20260925 16:33:22.420331 7813 ts_tablet_manager.cc:1905] T 21362dbfeedd4a6c908232ce6326aac6 P 37e3504cf5f848cc912031815590b8a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:33:22.423282 7813 ts_tablet_manager.cc:1918] T 21362dbfeedd4a6c908232ce6326aac6 P 37e3504cf5f848cc912031815590b8a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:33:22.423448 7813 log.cc:1196] T 21362dbfeedd4a6c908232ce6326aac6 P 37e3504cf5f848cc912031815590b8a8: Deleting WAL directory at /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-0/wal/wals/21362dbfeedd4a6c908232ce6326aac6
I20260925 16:33:22.423825 7813 ts_tablet_manager.cc:1939] T 21362dbfeedd4a6c908232ce6326aac6 P 37e3504cf5f848cc912031815590b8a8: Deleting consensus metadata
I20260925 16:33:22.420259 7815 tablet_replica.cc:330] T 21362dbfeedd4a6c908232ce6326aac6 P 0e11f3b8048a40d093ebd83dfdfe0b46: stopping tablet replica
I20260925 16:33:22.425870 7815 raft_consensus.cc:2229] T 21362dbfeedd4a6c908232ce6326aac6 P 0e11f3b8048a40d093ebd83dfdfe0b46 [term 1 LEADER]: Raft consensus shutting down.
I20260925 16:33:22.426258 7815 raft_consensus.cc:2258] T 21362dbfeedd4a6c908232ce6326aac6 P 0e11f3b8048a40d093ebd83dfdfe0b46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:33:22.427824 7814 ts_tablet_manager.cc:1905] T 21362dbfeedd4a6c908232ce6326aac6 P 8d93ff6ea52e43f295aa3c9846e05303: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 16:33:22.429328 7373 connection.cc:535] server connection from 127.0.0.1:38126 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 16:33:22.431185 7814 ts_tablet_manager.cc:1918] T 21362dbfeedd4a6c908232ce6326aac6 P 8d93ff6ea52e43f295aa3c9846e05303: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:33:22.431350 7814 log.cc:1196] T 21362dbfeedd4a6c908232ce6326aac6 P 8d93ff6ea52e43f295aa3c9846e05303: Deleting WAL directory at /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-2/wal/wals/21362dbfeedd4a6c908232ce6326aac6
I20260925 16:33:22.431721 7814 ts_tablet_manager.cc:1939] T 21362dbfeedd4a6c908232ce6326aac6 P 8d93ff6ea52e43f295aa3c9846e05303: Deleting consensus metadata
I20260925 16:33:22.432785 7769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:22.419021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41576 (local address 127.0.98.61:45951)
0925 16:33:22.419115 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:22.419119 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:22.420513 (+ 1394us) server_negotiation.cc:407] Connection header received
0925 16:33:22.420562 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:22.420567 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:22.420626 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:22.420704 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:22.422102 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:22.423029 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:22.424421 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:22.424878 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:22.424974 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:22.428337 (+ 3363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:22.428343 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:22.428350 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:22.428383 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:22.430492 (+ 2109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:22.431363 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:22.431370 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:22.431373 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:22.431472 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:22.431800 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:22.431803 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:22.431805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:22.432011 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:22.432125 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:22.432456 (+ 331us) server_negotiation.cc:299] Negotiation successful
0925 16:33:22.432606 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 16:33:22.432821 7646 connection.cc:441] server connection from 127.0.0.1:57112 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57112 (request call id 1) could send its response
I20260925 16:33:22.433521 7815 ts_tablet_manager.cc:1905] T 21362dbfeedd4a6c908232ce6326aac6 P 0e11f3b8048a40d093ebd83dfdfe0b46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:33:22.436386 7815 ts_tablet_manager.cc:1918] T 21362dbfeedd4a6c908232ce6326aac6 P 0e11f3b8048a40d093ebd83dfdfe0b46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:33:22.436482 7815 log.cc:1196] T 21362dbfeedd4a6c908232ce6326aac6 P 0e11f3b8048a40d093ebd83dfdfe0b46: Deleting WAL directory at /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-1/wal/wals/21362dbfeedd4a6c908232ce6326aac6
I20260925 16:33:22.436797 7815 ts_tablet_manager.cc:1939] T 21362dbfeedd4a6c908232ce6326aac6 P 0e11f3b8048a40d093ebd83dfdfe0b46: Deleting consensus metadata
W20260925 16:33:22.437845 7509 connection.cc:441] server connection from 127.0.0.1:45946 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:45946 (request call id 1) could send its response
I20260925 16:33:22.444092 7773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:22.419114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53566 (local address 127.0.98.60:37381)
0925 16:33:22.419201 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:22.419205 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:22.433647 (+ 14442us) server_negotiation.cc:407] Connection header received
0925 16:33:22.433826 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:22.433829 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:22.433884 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:22.433943 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:22.434902 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:22.435671 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:22.437317 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:22.437675 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:22.437773 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:22.439698 (+ 1925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:22.439702 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:22.439705 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:22.439730 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:22.441420 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:22.442192 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:22.442196 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:22.442198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:22.442287 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:22.442612 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:22.442616 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:22.442619 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:22.443126 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:22.443250 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:22.443760 (+ 510us) server_negotiation.cc:299] Negotiation successful
0925 16:33:22.443924 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 16:33:22.446812 7769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:22.435971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41586 (local address 127.0.98.61:45951)
0925 16:33:22.436136 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:22.436140 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:22.437689 (+ 1549us) server_negotiation.cc:407] Connection header received
0925 16:33:22.437881 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:22.437884 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:22.437930 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:22.437980 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:22.439138 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:22.439910 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:22.441036 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:22.441359 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:22.441448 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:22.443106 (+ 1658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:22.443112 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:22.443114 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:22.443139 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:22.444870 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:22.445372 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:22.445375 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:22.445377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:22.445462 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:22.445788 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:22.445793 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:22.445795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:22.446004 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:22.446124 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:22.446480 (+ 356us) server_negotiation.cc:299] Negotiation successful
0925 16:33:22.446634 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260925 16:33:22.447975 7491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:22.436093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53580 (local address 127.0.98.60:37381)
0925 16:33:22.436220 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:22.436227 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:22.438302 (+ 2075us) server_negotiation.cc:407] Connection header received
0925 16:33:22.438479 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:22.438483 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:22.438535 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:22.438723 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:22.439668 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:22.440394 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:22.441823 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:22.442157 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:22.442239 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:22.444184 (+ 1945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:22.444190 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:22.444193 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:22.444220 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:22.446209 (+ 1989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:22.446708 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:22.446712 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:22.446714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:22.446791 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:22.447080 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:22.447100 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:22.447102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:22.447285 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:22.447380 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:22.447705 (+ 325us) server_negotiation.cc:299] Negotiation successful
0925 16:33:22.447789 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45,"spinlock_wait_cycles":3456}
I20260925 16:33:23.405675 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:23.405194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34575 (local address 127.0.98.2:52435)
0925 16:33:23.405471 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:23.405574 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":89,"threads_started":1}
W20260925 16:33:23.405999 7622 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34575 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:34575: connect: Connection refused (error 111)
I20260925 16:33:23.406395 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:23.405916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34575 (local address 127.0.98.1:56647)
0925 16:33:23.406195 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:23.406278 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":65,"threads_started":1}
W20260925 16:33:23.406997 7485 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34575 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:34575: connect: Connection refused (error 111)
I20260925 16:33:23.408031 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:23.407588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34575 (local address 127.0.98.3:57279)
0925 16:33:23.407825 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:23.407921 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":77,"threads_started":1}
W20260925 16:33:23.408283 7759 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34575 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:34575: connect: Connection refused (error 111)
I20260925 16:33:23.916975 7827 raft_consensus.cc:482] T 00000000000000000000000000000000 P fffc9052c3ea4c7da2a350f0f2ca6d75 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6772881645114340bec15e5a7e09e6e5)
I20260925 16:33:23.917261 7828 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6772881645114340bec15e5a7e09e6e5)
I20260925 16:33:23.917240 7827 raft_consensus.cc:504] T 00000000000000000000000000000000 P fffc9052c3ea4c7da2a350f0f2ca6d75 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6772881645114340bec15e5a7e09e6e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34575 } } peers { permanent_uuid: "9fbd60470ceb46e2bb9f413b41a66ddb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45951 } attrs { promote: false } } peers { permanent_uuid: "fffc9052c3ea4c7da2a350f0f2ca6d75" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37381 } attrs { promote: false } }
I20260925 16:33:23.917450 7828 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6772881645114340bec15e5a7e09e6e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34575 } } peers { permanent_uuid: "9fbd60470ceb46e2bb9f413b41a66ddb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45951 } attrs { promote: false } } peers { permanent_uuid: "fffc9052c3ea4c7da2a350f0f2ca6d75" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37381 } attrs { promote: false } }
I20260925 16:33:23.918318 7828 leader_election.cc:288] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6772881645114340bec15e5a7e09e6e5 (127.0.98.62:34575), fffc9052c3ea4c7da2a350f0f2ca6d75 (127.0.98.60:37381)
I20260925 16:33:23.918362 7827 leader_election.cc:288] T 00000000000000000000000000000000 P fffc9052c3ea4c7da2a350f0f2ca6d75 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6772881645114340bec15e5a7e09e6e5 (127.0.98.62:34575), 9fbd60470ceb46e2bb9f413b41a66ddb (127.0.98.61:45951)
I20260925 16:33:23.918903 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:23.918448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34575 (local address 127.0.0.1:35164)
0925 16:33:23.918708 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:23.918789 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":63,"threads_started":1}
I20260925 16:33:23.919180 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:23.918803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34575 (local address 127.0.0.1:35162)
0925 16:33:23.919007 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:23.919073 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":77,"threads_started":1}
W20260925 16:33:23.919160 7289 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34575: connect: Connection refused (error 111)
W20260925 16:33:23.919567 7104 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34575: connect: Connection refused (error 111)
I20260925 16:33:23.920198 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:23.919973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34575 (local address 127.0.0.1:35190)
0925 16:33:23.920084 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:23.920135 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 16:33:23.920296 7289 leader_election.cc:334] T 00000000000000000000000000000000 P fffc9052c3ea4c7da2a350f0f2ca6d75 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6772881645114340bec15e5a7e09e6e5 (127.0.98.62:34575): Network error: Client connection negotiation failed: client connection to 127.0.98.62:34575: connect: Connection refused (error 111)
I20260925 16:33:23.920938 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:23.920531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34575 (local address 127.0.0.1:35202)
0925 16:33:23.920815 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:23.920863 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":86,"threads_started":1}
W20260925 16:33:23.921069 7104 leader_election.cc:334] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6772881645114340bec15e5a7e09e6e5 (127.0.98.62:34575): Network error: Client connection negotiation failed: client connection to 127.0.98.62:34575: connect: Connection refused (error 111)
I20260925 16:33:23.922936 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:23.918557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45951 (local address 127.0.0.1:41594)
0925 16:33:23.918902 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:23.918918 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:23.919071 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:23.919473 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:23.919476 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:23.919495 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:33:23.919609 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:23.919615 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:23.920659 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:23.920661 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:23.921875 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:23.921886 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:23.922515 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:23.922521 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:23.922635 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:23.922720 (+ 85us) client_negotiation.cc:769] Sending connection context
0925 16:33:23.922783 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 16:33:23.922838 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"mutex_wait_us":53,"thread_start_us":63,"threads_started":1}
I20260925 16:33:23.923002 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:23.918657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41594 (local address 127.0.98.61:45951)
0925 16:33:23.918885 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:23.918889 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:23.918997 (+ 108us) server_negotiation.cc:407] Connection header received
0925 16:33:23.919196 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:23.919202 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:23.919251 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:23.919309 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:33:23.919730 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:23.920547 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:23.922013 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:23.922400 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:23.922512 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:23.922573 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:23.922820 (+ 247us) server_negotiation.cc:299] Negotiation successful
0925 16:33:23.922889 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":67,"threads_started":1}
I20260925 16:33:23.923412 7130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fffc9052c3ea4c7da2a350f0f2ca6d75" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9fbd60470ceb46e2bb9f413b41a66ddb" is_pre_election: true
I20260925 16:33:23.923647 7130 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fffc9052c3ea4c7da2a350f0f2ca6d75 in term 1.
I20260925 16:33:23.923763 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:23.919398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37381 (local address 127.0.0.1:53594)
0925 16:33:23.919511 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:23.919523 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:23.919591 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:23.920292 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:23.920295 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:23.920312 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:33:23.920399 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:23.920406 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:23.921453 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:23.921457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:23.922391 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:23.922399 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:23.923348 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:23.923351 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:23.923435 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:23.923463 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 16:33:23.923543 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 16:33:23.923629 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 16:33:23.923777 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:23.919234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53594 (local address 127.0.98.60:37381)
0925 16:33:23.919942 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:23.919946 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:23.919960 (+ 14us) server_negotiation.cc:407] Connection header received
0925 16:33:23.920011 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:23.920014 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:23.920057 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:23.920140 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:33:23.920528 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:23.921285 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:23.922540 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:23.923255 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:23.923341 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:23.923390 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:23.923589 (+ 199us) server_negotiation.cc:299] Negotiation successful
0925 16:33:23.923665 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":57,"threads_started":1}
I20260925 16:33:23.924096 7292 leader_election.cc:302] T 00000000000000000000000000000000 P fffc9052c3ea4c7da2a350f0f2ca6d75 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9fbd60470ceb46e2bb9f413b41a66ddb, fffc9052c3ea4c7da2a350f0f2ca6d75; no voters: 6772881645114340bec15e5a7e09e6e5
I20260925 16:33:23.924124 7314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9fbd60470ceb46e2bb9f413b41a66ddb" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fffc9052c3ea4c7da2a350f0f2ca6d75" is_pre_election: true
I20260925 16:33:23.924310 7314 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fffc9052c3ea4c7da2a350f0f2ca6d75 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9fbd60470ceb46e2bb9f413b41a66ddb in term 1.
I20260925 16:33:23.924387 7827 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fffc9052c3ea4c7da2a350f0f2ca6d75 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 16:33:23.924440 7827 raft_consensus.cc:482] T 00000000000000000000000000000000 P fffc9052c3ea4c7da2a350f0f2ca6d75 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6772881645114340bec15e5a7e09e6e5)
I20260925 16:33:23.924502 7827 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fffc9052c3ea4c7da2a350f0f2ca6d75 [term 1 FOLLOWER]: Advancing to term 2
I20260925 16:33:23.924597 7104 leader_election.cc:302] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9fbd60470ceb46e2bb9f413b41a66ddb, fffc9052c3ea4c7da2a350f0f2ca6d75; no voters: 6772881645114340bec15e5a7e09e6e5
I20260925 16:33:23.924774 7828 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 16:33:23.924820 7828 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6772881645114340bec15e5a7e09e6e5)
I20260925 16:33:23.924857 7828 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb [term 1 FOLLOWER]: Advancing to term 2
I20260925 16:33:23.925500 7827 raft_consensus.cc:504] T 00000000000000000000000000000000 P fffc9052c3ea4c7da2a350f0f2ca6d75 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6772881645114340bec15e5a7e09e6e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34575 } } peers { permanent_uuid: "9fbd60470ceb46e2bb9f413b41a66ddb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45951 } attrs { promote: false } } peers { permanent_uuid: "fffc9052c3ea4c7da2a350f0f2ca6d75" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37381 } attrs { promote: false } }
I20260925 16:33:23.925750 7828 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6772881645114340bec15e5a7e09e6e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34575 } } peers { permanent_uuid: "9fbd60470ceb46e2bb9f413b41a66ddb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45951 } attrs { promote: false } } peers { permanent_uuid: "fffc9052c3ea4c7da2a350f0f2ca6d75" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37381 } attrs { promote: false } }
I20260925 16:33:23.925836 7827 leader_election.cc:288] T 00000000000000000000000000000000 P fffc9052c3ea4c7da2a350f0f2ca6d75 [CANDIDATE]: Term 2 election: Requested vote from peers 6772881645114340bec15e5a7e09e6e5 (127.0.98.62:34575), 9fbd60470ceb46e2bb9f413b41a66ddb (127.0.98.61:45951)
I20260925 16:33:23.926123 7828 leader_election.cc:288] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb [CANDIDATE]: Term 2 election: Requested vote from peers 6772881645114340bec15e5a7e09e6e5 (127.0.98.62:34575), fffc9052c3ea4c7da2a350f0f2ca6d75 (127.0.98.60:37381)
I20260925 16:33:23.926174 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:23.925901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34575 (local address 127.0.0.1:35214)
0925 16:33:23.926055 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:23.926111 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260925 16:33:23.926154 7130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fffc9052c3ea4c7da2a350f0f2ca6d75" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9fbd60470ceb46e2bb9f413b41a66ddb"
I20260925 16:33:23.926293 7130 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate fffc9052c3ea4c7da2a350f0f2ca6d75 in current term 2: Already voted for candidate 9fbd60470ceb46e2bb9f413b41a66ddb in this term.
I20260925 16:33:23.926383 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:23.926186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34575 (local address 127.0.0.1:35216)
0925 16:33:23.926275 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:23.926325 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260925 16:33:23.926795 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:23.926601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34575 (local address 127.0.0.1:35222)
0925 16:33:23.926688 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:23.926736 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260925 16:33:23.927006 7289 leader_election.cc:334] T 00000000000000000000000000000000 P fffc9052c3ea4c7da2a350f0f2ca6d75 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6772881645114340bec15e5a7e09e6e5 (127.0.98.62:34575): Network error: Client connection negotiation failed: client connection to 127.0.98.62:34575: connect: Connection refused (error 111)
I20260925 16:33:23.927039 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:23.926812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34575 (local address 127.0.0.1:35228)
0925 16:33:23.926933 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:23.926986 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 16:33:23.927089 7289 leader_election.cc:302] T 00000000000000000000000000000000 P fffc9052c3ea4c7da2a350f0f2ca6d75 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fffc9052c3ea4c7da2a350f0f2ca6d75; no voters: 6772881645114340bec15e5a7e09e6e5, 9fbd60470ceb46e2bb9f413b41a66ddb
W20260925 16:33:23.927148 7104 leader_election.cc:334] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6772881645114340bec15e5a7e09e6e5 (127.0.98.62:34575): Network error: Client connection negotiation failed: client connection to 127.0.98.62:34575: connect: Connection refused (error 111)
I20260925 16:33:23.927217 7827 raft_consensus.cc:2726] T 00000000000000000000000000000000 P fffc9052c3ea4c7da2a350f0f2ca6d75 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 16:33:23.926846 7314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9fbd60470ceb46e2bb9f413b41a66ddb" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fffc9052c3ea4c7da2a350f0f2ca6d75"
I20260925 16:33:23.927479 7314 raft_consensus.cc:2379] T 00000000000000000000000000000000 P fffc9052c3ea4c7da2a350f0f2ca6d75 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 9fbd60470ceb46e2bb9f413b41a66ddb in current term 2: Already voted for candidate fffc9052c3ea4c7da2a350f0f2ca6d75 in this term.
I20260925 16:33:23.927819 7104 leader_election.cc:302] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9fbd60470ceb46e2bb9f413b41a66ddb; no voters: 6772881645114340bec15e5a7e09e6e5, fffc9052c3ea4c7da2a350f0f2ca6d75
I20260925 16:33:23.928020 7828 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 16:33:24.407068 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:24.406564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34575 (local address 127.0.98.2:38829)
0925 16:33:24.406915 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:24.406967 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":95,"threads_started":1}
I20260925 16:33:24.407997 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:24.407594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34575 (local address 127.0.98.1:40151)
0925 16:33:24.407827 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:24.407888 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":62,"threads_started":1}
I20260925 16:33:24.409135 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:24.408726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34575 (local address 127.0.98.3:36999)
0925 16:33:24.408964 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:24.409039 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":59,"threads_started":1}
I20260925 16:33:25.408181 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:25.407775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34575 (local address 127.0.98.2:38451)
0925 16:33:25.408031 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:25.408092 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":72,"threads_started":1}
I20260925 16:33:25.409708 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:25.409005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34575 (local address 127.0.98.1:39521)
0925 16:33:25.409538 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:25.409608 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":90,"threads_started":1}
I20260925 16:33:25.410270 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:25.409735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34575 (local address 127.0.98.3:44587)
0925 16:33:25.410108 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:25.410164 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":66,"threads_started":1}
I20260925 16:33:25.788194 7843 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:33:25.788357 7843 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6772881645114340bec15e5a7e09e6e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34575 } } peers { permanent_uuid: "9fbd60470ceb46e2bb9f413b41a66ddb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45951 } attrs { promote: false } } peers { permanent_uuid: "fffc9052c3ea4c7da2a350f0f2ca6d75" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37381 } attrs { promote: false } }
I20260925 16:33:25.788944 7843 leader_election.cc:288] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6772881645114340bec15e5a7e09e6e5 (127.0.98.62:34575), fffc9052c3ea4c7da2a350f0f2ca6d75 (127.0.98.60:37381)
I20260925 16:33:25.789525 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:25.789116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34575 (local address 127.0.0.1:43970)
0925 16:33:25.789346 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:25.789419 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":82,"threads_started":1}
I20260925 16:33:25.789705 7314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9fbd60470ceb46e2bb9f413b41a66ddb" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fffc9052c3ea4c7da2a350f0f2ca6d75" is_pre_election: true
I20260925 16:33:25.789855 7314 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fffc9052c3ea4c7da2a350f0f2ca6d75 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9fbd60470ceb46e2bb9f413b41a66ddb in term 2.
I20260925 16:33:25.790385 7104 leader_election.cc:302] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9fbd60470ceb46e2bb9f413b41a66ddb, fffc9052c3ea4c7da2a350f0f2ca6d75; no voters:
I20260925 16:33:25.790482 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:25.790234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34575 (local address 127.0.0.1:43986)
0925 16:33:25.790340 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:25.790387 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 16:33:25.790557 7104 leader_election.cc:334] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6772881645114340bec15e5a7e09e6e5 (127.0.98.62:34575): Network error: Client connection negotiation failed: client connection to 127.0.98.62:34575: connect: Connection refused (error 111)
I20260925 16:33:25.790622 7843 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 16:33:25.790699 7843 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:33:25.790768 7843 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb [term 2 FOLLOWER]: Advancing to term 3
I20260925 16:33:25.791777 7843 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6772881645114340bec15e5a7e09e6e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34575 } } peers { permanent_uuid: "9fbd60470ceb46e2bb9f413b41a66ddb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45951 } attrs { promote: false } } peers { permanent_uuid: "fffc9052c3ea4c7da2a350f0f2ca6d75" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37381 } attrs { promote: false } }
I20260925 16:33:25.792086 7843 leader_election.cc:288] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb [CANDIDATE]: Term 3 election: Requested vote from peers 6772881645114340bec15e5a7e09e6e5 (127.0.98.62:34575), fffc9052c3ea4c7da2a350f0f2ca6d75 (127.0.98.60:37381)
I20260925 16:33:25.792407 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:25.792217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34575 (local address 127.0.0.1:44000)
0925 16:33:25.792296 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:25.792348 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260925 16:33:25.792598 7314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9fbd60470ceb46e2bb9f413b41a66ddb" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fffc9052c3ea4c7da2a350f0f2ca6d75"
I20260925 16:33:25.792718 7314 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fffc9052c3ea4c7da2a350f0f2ca6d75 [term 2 FOLLOWER]: Advancing to term 3
I20260925 16:33:25.793046 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:25.792853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34575 (local address 127.0.0.1:44004)
0925 16:33:25.792953 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:25.792996 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 16:33:25.793144 7104 leader_election.cc:334] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 6772881645114340bec15e5a7e09e6e5 (127.0.98.62:34575): Network error: Client connection negotiation failed: client connection to 127.0.98.62:34575: connect: Connection refused (error 111)
I20260925 16:33:25.793689 7314 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fffc9052c3ea4c7da2a350f0f2ca6d75 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fbd60470ceb46e2bb9f413b41a66ddb in term 3.
I20260925 16:33:25.793967 7104 leader_election.cc:302] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9fbd60470ceb46e2bb9f413b41a66ddb, fffc9052c3ea4c7da2a350f0f2ca6d75; no voters: 6772881645114340bec15e5a7e09e6e5
I20260925 16:33:25.794135 7843 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb [term 3 FOLLOWER]: Leader election won for term 3
I20260925 16:33:25.794334 7843 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb [term 3 LEADER]: Becoming Leader. State: Replica: 9fbd60470ceb46e2bb9f413b41a66ddb, State: Running, Role: LEADER
I20260925 16:33:25.794633 7843 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6772881645114340bec15e5a7e09e6e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34575 } } peers { permanent_uuid: "9fbd60470ceb46e2bb9f413b41a66ddb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45951 } attrs { promote: false } } peers { permanent_uuid: "fffc9052c3ea4c7da2a350f0f2ca6d75" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37381 } attrs { promote: false } }
I20260925 16:33:25.795964 7846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fbd60470ceb46e2bb9f413b41a66ddb. Latest consensus state: current_term: 3 leader_uuid: "9fbd60470ceb46e2bb9f413b41a66ddb" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6772881645114340bec15e5a7e09e6e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34575 } } peers { permanent_uuid: "9fbd60470ceb46e2bb9f413b41a66ddb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45951 } attrs { promote: false } } peers { permanent_uuid: "fffc9052c3ea4c7da2a350f0f2ca6d75" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37381 } attrs { promote: false } } }
I20260925 16:33:25.796069 7846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb [sys.catalog]: This master's current role is: LEADER
I20260925 16:33:25.796303 7848 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:33:25.798472 7848 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:33:25.798890 7848 catalog_manager.cc:1259] Loaded cluster ID: 05ab274724e944f197784d0f601636d0
I20260925 16:33:25.798933 7848 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:33:25.799472 7848 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:33:25.799857 7848 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb: Loaded TSK: 0
I20260925 16:33:25.800539 7848 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:33:25.826639 7120 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41586:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 16:33:25.829617 7120 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:33:25.832701 7314 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fffc9052c3ea4c7da2a350f0f2ca6d75 [term 3 FOLLOWER]: Refusing update from remote peer 9fbd60470ceb46e2bb9f413b41a66ddb: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260925 16:33:25.832720 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:25.832486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34575 (local address 127.0.0.1:44018)
0925 16:33:25.832613 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:25.832657 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 16:33:25.833155 7846 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fffc9052c3ea4c7da2a350f0f2ca6d75" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260925 16:33:25.833508 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:25.833276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34575 (local address 127.0.0.1:44020)
0925 16:33:25.833398 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:25.833455 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 16:33:25.833679 7104 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb -> Peer 6772881645114340bec15e5a7e09e6e5 (127.0.98.62:34575): Couldn't send request to peer 6772881645114340bec15e5a7e09e6e5. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34575: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 16:33:25.834610 7850 sys_catalog.cc:455] T 00000000000000000000000000000000 P fffc9052c3ea4c7da2a350f0f2ca6d75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fbd60470ceb46e2bb9f413b41a66ddb. Latest consensus state: current_term: 3 leader_uuid: "9fbd60470ceb46e2bb9f413b41a66ddb" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6772881645114340bec15e5a7e09e6e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34575 } } peers { permanent_uuid: "9fbd60470ceb46e2bb9f413b41a66ddb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45951 } attrs { promote: false } } peers { permanent_uuid: "fffc9052c3ea4c7da2a350f0f2ca6d75" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37381 } attrs { promote: false } } }
I20260925 16:33:25.834748 7850 sys_catalog.cc:458] T 00000000000000000000000000000000 P fffc9052c3ea4c7da2a350f0f2ca6d75 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:33:25.835978 7846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "9fbd60470ceb46e2bb9f413b41a66ddb" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6772881645114340bec15e5a7e09e6e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34575 } } peers { permanent_uuid: "9fbd60470ceb46e2bb9f413b41a66ddb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45951 } attrs { promote: false } } peers { permanent_uuid: "fffc9052c3ea4c7da2a350f0f2ca6d75" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37381 } attrs { promote: false } } }
I20260925 16:33:25.836097 7846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb [sys.catalog]: This master's current role is: LEADER
I20260925 16:33:25.836418 7850 sys_catalog.cc:455] T 00000000000000000000000000000000 P fffc9052c3ea4c7da2a350f0f2ca6d75 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "9fbd60470ceb46e2bb9f413b41a66ddb" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6772881645114340bec15e5a7e09e6e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34575 } } peers { permanent_uuid: "9fbd60470ceb46e2bb9f413b41a66ddb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45951 } attrs { promote: false } } peers { permanent_uuid: "fffc9052c3ea4c7da2a350f0f2ca6d75" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37381 } attrs { promote: false } } }
I20260925 16:33:25.836540 7850 sys_catalog.cc:458] T 00000000000000000000000000000000 P fffc9052c3ea4c7da2a350f0f2ca6d75 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:33:25.836764 7851 mvcc.cc:205] Tried to move back new op lower bound from 7333290007885737984 to 7333290007736967168. Current Snapshot: MvccSnapshot[applied={T|T < 7333290007885737984}]
I20260925 16:33:25.847434 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:25.842368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43771 (local address 127.0.0.1:40406)
0925 16:33:25.842738 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:25.842750 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:25.842830 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:25.843575 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:25.843578 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:25.843594 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:33:25.843702 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:25.843707 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:25.844685 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:25.844688 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:25.845803 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:25.845812 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:25.846984 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:25.846987 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:25.847092 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:25.847118 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 16:33:25.847192 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 16:33:25.847277 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"mutex_wait_us":39,"thread_start_us":81,"threads_started":1}
I20260925 16:33:25.847851 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:25.842479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40406 (local address 127.0.98.2:43771)
0925 16:33:25.843337 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:25.843347 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:25.843359 (+ 12us) server_negotiation.cc:407] Connection header received
0925 16:33:25.843406 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:25.843408 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:25.843477 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:25.843567 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:33:25.843788 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:25.844572 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:25.845933 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:25.846752 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:25.846859 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:25.846925 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:25.847603 (+ 678us) server_negotiation.cc:299] Negotiation successful
0925 16:33:25.847686 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":784,"thread_start_us":97,"threads_started":1}
I20260925 16:33:25.847899 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:25.841889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43207 (local address 127.0.0.1:43226)
0925 16:33:25.842000 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:25.842015 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:25.842127 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:25.842548 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:25.842554 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:25.842608 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:33:25.842723 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:25.842730 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:25.843867 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:25.843873 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:25.844772 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:25.844780 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:25.847216 (+ 2436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:25.847219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:25.847318 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:25.847347 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 16:33:25.847415 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 16:33:25.847780 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260925 16:33:25.848343 7860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:25.842780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48120 (local address 127.0.98.3:42375)
0925 16:33:25.843173 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:25.843177 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:25.843191 (+ 14us) server_negotiation.cc:407] Connection header received
0925 16:33:25.843307 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:25.843311 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:25.843359 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:25.843444 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:33:25.843784 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:25.844572 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:25.846306 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:25.847225 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:25.847343 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:25.847427 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:25.847860 (+ 433us) server_negotiation.cc:299] Negotiation successful
0925 16:33:25.848217 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":61,"threads_started":1}
I20260925 16:33:25.848480 7555 tablet_service.cc:1467] Processing CreateTablet for tablet c87cdec4d1ca41098090cb9e6eda5ca2 (DEFAULT_TABLE table=test-table [id=9c2aaeeda98546bd91e186ac257f7a15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:33:25.848702 7856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:25.842006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43226 (local address 127.0.98.1:43207)
0925 16:33:25.842261 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:25.842264 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:25.842277 (+ 13us) server_negotiation.cc:407] Connection header received
0925 16:33:25.842346 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:25.842349 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:25.842403 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:25.842846 (+ 443us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:33:25.842953 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:25.843787 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:25.844871 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:25.847073 (+ 2202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:25.848271 (+ 1198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:25.848342 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:25.848379 (+ 37us) server_negotiation.cc:299] Negotiation successful
0925 16:33:25.848528 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":76,"threads_started":1}
I20260925 16:33:25.848871 7555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c87cdec4d1ca41098090cb9e6eda5ca2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:33:25.849283 7418 tablet_service.cc:1467] Processing CreateTablet for tablet c87cdec4d1ca41098090cb9e6eda5ca2 (DEFAULT_TABLE table=test-table [id=9c2aaeeda98546bd91e186ac257f7a15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:33:25.849530 7418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c87cdec4d1ca41098090cb9e6eda5ca2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:33:25.850895 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:25.842475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42375 (local address 127.0.0.1:48120)
0925 16:33:25.843089 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:25.843103 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:25.843187 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:25.843469 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:25.843472 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:25.843487 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:33:25.843658 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:25.843665 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:25.844723 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:25.844726 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:25.846064 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:25.846075 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:25.847520 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:25.847525 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:25.847659 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:25.847680 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 16:33:25.847800 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 16:33:25.850805 (+ 3005us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"mutex_wait_us":9,"thread_start_us":73,"threads_started":1}
I20260925 16:33:25.851526 7861 tablet_bootstrap.cc:492] T c87cdec4d1ca41098090cb9e6eda5ca2 P 0e11f3b8048a40d093ebd83dfdfe0b46: Bootstrap starting.
I20260925 16:33:25.851487 7692 tablet_service.cc:1467] Processing CreateTablet for tablet c87cdec4d1ca41098090cb9e6eda5ca2 (DEFAULT_TABLE table=test-table [id=9c2aaeeda98546bd91e186ac257f7a15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:33:25.851740 7692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c87cdec4d1ca41098090cb9e6eda5ca2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:33:25.852334 7862 tablet_bootstrap.cc:492] T c87cdec4d1ca41098090cb9e6eda5ca2 P 37e3504cf5f848cc912031815590b8a8: Bootstrap starting.
I20260925 16:33:25.852494 7861 tablet_bootstrap.cc:654] T c87cdec4d1ca41098090cb9e6eda5ca2 P 0e11f3b8048a40d093ebd83dfdfe0b46: Neither blocks nor log segments found. Creating new log.
I20260925 16:33:25.853104 7862 tablet_bootstrap.cc:654] T c87cdec4d1ca41098090cb9e6eda5ca2 P 37e3504cf5f848cc912031815590b8a8: Neither blocks nor log segments found. Creating new log.
I20260925 16:33:25.853609 7861 tablet_bootstrap.cc:492] T c87cdec4d1ca41098090cb9e6eda5ca2 P 0e11f3b8048a40d093ebd83dfdfe0b46: No bootstrap required, opened a new log
I20260925 16:33:25.853696 7861 ts_tablet_manager.cc:1397] T c87cdec4d1ca41098090cb9e6eda5ca2 P 0e11f3b8048a40d093ebd83dfdfe0b46: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 16:33:25.854239 7862 tablet_bootstrap.cc:492] T c87cdec4d1ca41098090cb9e6eda5ca2 P 37e3504cf5f848cc912031815590b8a8: No bootstrap required, opened a new log
I20260925 16:33:25.854245 7865 tablet_bootstrap.cc:492] T c87cdec4d1ca41098090cb9e6eda5ca2 P 8d93ff6ea52e43f295aa3c9846e05303: Bootstrap starting.
I20260925 16:33:25.854167 7861 raft_consensus.cc:348] T c87cdec4d1ca41098090cb9e6eda5ca2 P 0e11f3b8048a40d093ebd83dfdfe0b46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37e3504cf5f848cc912031815590b8a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43207 } } peers { permanent_uuid: "0e11f3b8048a40d093ebd83dfdfe0b46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43771 } } peers { permanent_uuid: "8d93ff6ea52e43f295aa3c9846e05303" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42375 } }
I20260925 16:33:25.854560 7861 raft_consensus.cc:374] T c87cdec4d1ca41098090cb9e6eda5ca2 P 0e11f3b8048a40d093ebd83dfdfe0b46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:33:25.854648 7861 raft_consensus.cc:729] T c87cdec4d1ca41098090cb9e6eda5ca2 P 0e11f3b8048a40d093ebd83dfdfe0b46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e11f3b8048a40d093ebd83dfdfe0b46, State: Initialized, Role: FOLLOWER
I20260925 16:33:25.854794 7861 consensus_queue.cc:260] T c87cdec4d1ca41098090cb9e6eda5ca2 P 0e11f3b8048a40d093ebd83dfdfe0b46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37e3504cf5f848cc912031815590b8a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43207 } } peers { permanent_uuid: "0e11f3b8048a40d093ebd83dfdfe0b46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43771 } } peers { permanent_uuid: "8d93ff6ea52e43f295aa3c9846e05303" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42375 } }
I20260925 16:33:25.854912 7862 ts_tablet_manager.cc:1397] T c87cdec4d1ca41098090cb9e6eda5ca2 P 37e3504cf5f848cc912031815590b8a8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 16:33:25.855146 7865 tablet_bootstrap.cc:654] T c87cdec4d1ca41098090cb9e6eda5ca2 P 8d93ff6ea52e43f295aa3c9846e05303: Neither blocks nor log segments found. Creating new log.
I20260925 16:33:25.855167 7861 ts_tablet_manager.cc:1428] T c87cdec4d1ca41098090cb9e6eda5ca2 P 0e11f3b8048a40d093ebd83dfdfe0b46: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260925 16:33:25.855865 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:25.855615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34575 (local address 127.0.98.2:56385)
0925 16:33:25.855747 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:25.855805 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260925 16:33:25.856822 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:25.856584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34575 (local address 127.0.98.2:58391)
0925 16:33:25.856685 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:25.856763 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 16:33:25.860381 7623 heartbeater.cc:503] Master 127.0.98.61:45951 was elected leader, sending a full tablet report...
I20260925 16:33:25.860404 7862 raft_consensus.cc:348] T c87cdec4d1ca41098090cb9e6eda5ca2 P 37e3504cf5f848cc912031815590b8a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37e3504cf5f848cc912031815590b8a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43207 } } peers { permanent_uuid: "0e11f3b8048a40d093ebd83dfdfe0b46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43771 } } peers { permanent_uuid: "8d93ff6ea52e43f295aa3c9846e05303" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42375 } }
I20260925 16:33:25.860543 7862 raft_consensus.cc:374] T c87cdec4d1ca41098090cb9e6eda5ca2 P 37e3504cf5f848cc912031815590b8a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:33:25.860582 7862 raft_consensus.cc:729] T c87cdec4d1ca41098090cb9e6eda5ca2 P 37e3504cf5f848cc912031815590b8a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37e3504cf5f848cc912031815590b8a8, State: Initialized, Role: FOLLOWER
I20260925 16:33:25.860723 7862 consensus_queue.cc:260] T c87cdec4d1ca41098090cb9e6eda5ca2 P 37e3504cf5f848cc912031815590b8a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37e3504cf5f848cc912031815590b8a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43207 } } peers { permanent_uuid: "0e11f3b8048a40d093ebd83dfdfe0b46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43771 } } peers { permanent_uuid: "8d93ff6ea52e43f295aa3c9846e05303" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42375 } }
I20260925 16:33:25.861135 7862 ts_tablet_manager.cc:1428] T c87cdec4d1ca41098090cb9e6eda5ca2 P 37e3504cf5f848cc912031815590b8a8: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20260925 16:33:25.862025 7865 tablet_bootstrap.cc:492] T c87cdec4d1ca41098090cb9e6eda5ca2 P 8d93ff6ea52e43f295aa3c9846e05303: No bootstrap required, opened a new log
I20260925 16:33:25.862102 7865 ts_tablet_manager.cc:1397] T c87cdec4d1ca41098090cb9e6eda5ca2 P 8d93ff6ea52e43f295aa3c9846e05303: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260925 16:33:25.862260 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:25.861958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34575 (local address 127.0.98.1:47913)
0925 16:33:25.862142 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:25.862195 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260925 16:33:25.862498 7865 raft_consensus.cc:348] T c87cdec4d1ca41098090cb9e6eda5ca2 P 8d93ff6ea52e43f295aa3c9846e05303 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37e3504cf5f848cc912031815590b8a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43207 } } peers { permanent_uuid: "0e11f3b8048a40d093ebd83dfdfe0b46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43771 } } peers { permanent_uuid: "8d93ff6ea52e43f295aa3c9846e05303" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42375 } }
I20260925 16:33:25.862635 7865 raft_consensus.cc:374] T c87cdec4d1ca41098090cb9e6eda5ca2 P 8d93ff6ea52e43f295aa3c9846e05303 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:33:25.862673 7865 raft_consensus.cc:729] T c87cdec4d1ca41098090cb9e6eda5ca2 P 8d93ff6ea52e43f295aa3c9846e05303 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d93ff6ea52e43f295aa3c9846e05303, State: Initialized, Role: FOLLOWER
I20260925 16:33:25.862825 7865 consensus_queue.cc:260] T c87cdec4d1ca41098090cb9e6eda5ca2 P 8d93ff6ea52e43f295aa3c9846e05303 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37e3504cf5f848cc912031815590b8a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43207 } } peers { permanent_uuid: "0e11f3b8048a40d093ebd83dfdfe0b46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43771 } } peers { permanent_uuid: "8d93ff6ea52e43f295aa3c9846e05303" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42375 } }
I20260925 16:33:25.863214 7865 ts_tablet_manager.cc:1428] T c87cdec4d1ca41098090cb9e6eda5ca2 P 8d93ff6ea52e43f295aa3c9846e05303: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 16:33:25.863646 7486 heartbeater.cc:503] Master 127.0.98.61:45951 was elected leader, sending a full tablet report...
I20260925 16:33:25.863700 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:25.863469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34575 (local address 127.0.98.3:59699)
0925 16:33:25.863580 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:25.863633 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 16:33:25.864573 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:25.864311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34575 (local address 127.0.98.3:40891)
0925 16:33:25.864435 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:25.864505 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 16:33:25.864993 7760 heartbeater.cc:503] Master 127.0.98.61:45951 was elected leader, sending a full tablet report...
I20260925 16:33:25.866266 7866 raft_consensus.cc:482] T c87cdec4d1ca41098090cb9e6eda5ca2 P 0e11f3b8048a40d093ebd83dfdfe0b46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:33:25.866348 7866 raft_consensus.cc:504] T c87cdec4d1ca41098090cb9e6eda5ca2 P 0e11f3b8048a40d093ebd83dfdfe0b46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37e3504cf5f848cc912031815590b8a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43207 } } peers { permanent_uuid: "0e11f3b8048a40d093ebd83dfdfe0b46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43771 } } peers { permanent_uuid: "8d93ff6ea52e43f295aa3c9846e05303" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42375 } }
I20260925 16:33:25.866791 7866 leader_election.cc:288] T c87cdec4d1ca41098090cb9e6eda5ca2 P 0e11f3b8048a40d093ebd83dfdfe0b46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37e3504cf5f848cc912031815590b8a8 (127.0.98.1:43207), 8d93ff6ea52e43f295aa3c9846e05303 (127.0.98.3:42375)
I20260925 16:33:25.867090 7438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c87cdec4d1ca41098090cb9e6eda5ca2" candidate_uuid: "0e11f3b8048a40d093ebd83dfdfe0b46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37e3504cf5f848cc912031815590b8a8" is_pre_election: true
I20260925 16:33:25.867110 7712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c87cdec4d1ca41098090cb9e6eda5ca2" candidate_uuid: "0e11f3b8048a40d093ebd83dfdfe0b46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d93ff6ea52e43f295aa3c9846e05303" is_pre_election: true
I20260925 16:33:25.867223 7438 raft_consensus.cc:2454] T c87cdec4d1ca41098090cb9e6eda5ca2 P 37e3504cf5f848cc912031815590b8a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e11f3b8048a40d093ebd83dfdfe0b46 in term 0.
I20260925 16:33:25.867223 7712 raft_consensus.cc:2454] T c87cdec4d1ca41098090cb9e6eda5ca2 P 8d93ff6ea52e43f295aa3c9846e05303 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e11f3b8048a40d093ebd83dfdfe0b46 in term 0.
I20260925 16:33:25.867502 7510 leader_election.cc:302] T c87cdec4d1ca41098090cb9e6eda5ca2 P 0e11f3b8048a40d093ebd83dfdfe0b46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e11f3b8048a40d093ebd83dfdfe0b46, 8d93ff6ea52e43f295aa3c9846e05303; no voters:
I20260925 16:33:25.867653 7866 raft_consensus.cc:2781] T c87cdec4d1ca41098090cb9e6eda5ca2 P 0e11f3b8048a40d093ebd83dfdfe0b46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:33:25.867712 7866 raft_consensus.cc:482] T c87cdec4d1ca41098090cb9e6eda5ca2 P 0e11f3b8048a40d093ebd83dfdfe0b46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:33:25.867756 7866 raft_consensus.cc:3037] T c87cdec4d1ca41098090cb9e6eda5ca2 P 0e11f3b8048a40d093ebd83dfdfe0b46 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:33:25.868638 7866 raft_consensus.cc:504] T c87cdec4d1ca41098090cb9e6eda5ca2 P 0e11f3b8048a40d093ebd83dfdfe0b46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37e3504cf5f848cc912031815590b8a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43207 } } peers { permanent_uuid: "0e11f3b8048a40d093ebd83dfdfe0b46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43771 } } peers { permanent_uuid: "8d93ff6ea52e43f295aa3c9846e05303" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42375 } }
I20260925 16:33:25.868980 7866 leader_election.cc:288] T c87cdec4d1ca41098090cb9e6eda5ca2 P 0e11f3b8048a40d093ebd83dfdfe0b46 [CANDIDATE]: Term 1 election: Requested vote from peers 37e3504cf5f848cc912031815590b8a8 (127.0.98.1:43207), 8d93ff6ea52e43f295aa3c9846e05303 (127.0.98.3:42375)
I20260925 16:33:25.869251 7712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c87cdec4d1ca41098090cb9e6eda5ca2" candidate_uuid: "0e11f3b8048a40d093ebd83dfdfe0b46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d93ff6ea52e43f295aa3c9846e05303"
I20260925 16:33:25.869246 7438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c87cdec4d1ca41098090cb9e6eda5ca2" candidate_uuid: "0e11f3b8048a40d093ebd83dfdfe0b46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37e3504cf5f848cc912031815590b8a8"
I20260925 16:33:25.869355 7712 raft_consensus.cc:3037] T c87cdec4d1ca41098090cb9e6eda5ca2 P 8d93ff6ea52e43f295aa3c9846e05303 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:33:25.869355 7438 raft_consensus.cc:3037] T c87cdec4d1ca41098090cb9e6eda5ca2 P 37e3504cf5f848cc912031815590b8a8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:33:25.870371 7438 raft_consensus.cc:2454] T c87cdec4d1ca41098090cb9e6eda5ca2 P 37e3504cf5f848cc912031815590b8a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e11f3b8048a40d093ebd83dfdfe0b46 in term 1.
I20260925 16:33:25.870369 7712 raft_consensus.cc:2454] T c87cdec4d1ca41098090cb9e6eda5ca2 P 8d93ff6ea52e43f295aa3c9846e05303 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e11f3b8048a40d093ebd83dfdfe0b46 in term 1.
I20260925 16:33:25.870657 7510 leader_election.cc:302] T c87cdec4d1ca41098090cb9e6eda5ca2 P 0e11f3b8048a40d093ebd83dfdfe0b46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e11f3b8048a40d093ebd83dfdfe0b46, 8d93ff6ea52e43f295aa3c9846e05303; no voters:
I20260925 16:33:25.870803 7866 raft_consensus.cc:2781] T c87cdec4d1ca41098090cb9e6eda5ca2 P 0e11f3b8048a40d093ebd83dfdfe0b46 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:33:25.870999 7866 raft_consensus.cc:686] T c87cdec4d1ca41098090cb9e6eda5ca2 P 0e11f3b8048a40d093ebd83dfdfe0b46 [term 1 LEADER]: Becoming Leader. State: Replica: 0e11f3b8048a40d093ebd83dfdfe0b46, State: Running, Role: LEADER
I20260925 16:33:25.871142 7866 consensus_queue.cc:237] T c87cdec4d1ca41098090cb9e6eda5ca2 P 0e11f3b8048a40d093ebd83dfdfe0b46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37e3504cf5f848cc912031815590b8a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43207 } } peers { permanent_uuid: "0e11f3b8048a40d093ebd83dfdfe0b46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43771 } } peers { permanent_uuid: "8d93ff6ea52e43f295aa3c9846e05303" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42375 } }
I20260925 16:33:25.871757 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:25.871433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34575 (local address 127.0.98.2:57547)
0925 16:33:25.871621 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:25.871670 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260925 16:33:25.873044 7120 catalog_manager.cc:5495] T c87cdec4d1ca41098090cb9e6eda5ca2 P 0e11f3b8048a40d093ebd83dfdfe0b46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e11f3b8048a40d093ebd83dfdfe0b46 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "0e11f3b8048a40d093ebd83dfdfe0b46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37e3504cf5f848cc912031815590b8a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e11f3b8048a40d093ebd83dfdfe0b46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d93ff6ea52e43f295aa3c9846e05303" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42375 } health_report { overall_health: UNKNOWN } } }
I20260925 16:33:25.887331 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:25.884295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40414 (local address 127.0.98.2:43771)
0925 16:33:25.884422 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:25.884426 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:25.884440 (+ 14us) server_negotiation.cc:407] Connection header received
0925 16:33:25.884556 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:25.884558 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:25.884609 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:25.884683 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:33:25.884925 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:25.885417 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:25.886266 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:25.886490 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:25.886620 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:25.886885 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:33:25.886950 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:33:25.887051 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:25.887142 (+ 91us) server_negotiation.cc:299] Negotiation successful
0925 16:33:25.887194 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 16:33:25.889241 7438 raft_consensus.cc:1261] T c87cdec4d1ca41098090cb9e6eda5ca2 P 37e3504cf5f848cc912031815590b8a8 [term 1 FOLLOWER]: Refusing update from remote peer 0e11f3b8048a40d093ebd83dfdfe0b46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:33:25.889257 7712 raft_consensus.cc:1261] T c87cdec4d1ca41098090cb9e6eda5ca2 P 8d93ff6ea52e43f295aa3c9846e05303 [term 1 FOLLOWER]: Refusing update from remote peer 0e11f3b8048a40d093ebd83dfdfe0b46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:33:25.889674 7866 consensus_queue.cc:1035] T c87cdec4d1ca41098090cb9e6eda5ca2 P 0e11f3b8048a40d093ebd83dfdfe0b46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37e3504cf5f848cc912031815590b8a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:33:25.889806 7870 consensus_queue.cc:1035] T c87cdec4d1ca41098090cb9e6eda5ca2 P 0e11f3b8048a40d093ebd83dfdfe0b46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d93ff6ea52e43f295aa3c9846e05303" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:33:25.892037 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:25.891785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34575 (local address 127.0.98.3:59611)
0925 16:33:25.891905 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:25.891965 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 16:33:25.892377 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:25.891979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34575 (local address 127.0.98.1:40601)
0925 16:33:25.892261 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:25.892311 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218}
I20260925 16:33:25.894520 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:25.894289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34575 (local address 127.0.98.1:33019)
0925 16:33:25.894401 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:25.894450 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 16:33:25.894606 7875 mvcc.cc:205] Tried to move back new op lower bound from 7333290008118452224 to 7333290008049995776. Current Snapshot: MvccSnapshot[applied={T|T < 7333290008118452224}]
I20260925 16:33:25.896080 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:25.895808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34575 (local address 127.0.98.3:46191)
0925 16:33:25.895973 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:25.896018 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260925 16:33:25.896682 7873 mvcc.cc:205] Tried to move back new op lower bound from 7333290008118452224 to 7333290008049995776. Current Snapshot: MvccSnapshot[applied={T|T < 7333290008118452224 or (T in {7333290008118452224})}]
I20260925 16:33:25.897280 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:25.897035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34575 (local address 127.0.98.2:59707)
0925 16:33:25.897169 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:25.897217 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260925 16:33:25.899119 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:25.898875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34575 (local address 127.0.98.2:38389)
0925 16:33:25.899014 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:25.899062 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260925 16:33:25.899196 7119 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:33:25.899326 7119 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:33:25.902768 7119 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9fbd60470ceb46e2bb9f413b41a66ddb: Sending DeleteTablet for 3 replicas of tablet c87cdec4d1ca41098090cb9e6eda5ca2
I20260925 16:33:25.903496 7692 tablet_service.cc:1514] Processing DeleteTablet for tablet c87cdec4d1ca41098090cb9e6eda5ca2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:33:25 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:48120
I20260925 16:33:25.903497 7418 tablet_service.cc:1514] Processing DeleteTablet for tablet c87cdec4d1ca41098090cb9e6eda5ca2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:33:25 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:43226
I20260925 16:33:25.903841 7882 tablet_replica.cc:330] T c87cdec4d1ca41098090cb9e6eda5ca2 P 8d93ff6ea52e43f295aa3c9846e05303: stopping tablet replica
I20260925 16:33:25.903816 7555 tablet_service.cc:1514] Processing DeleteTablet for tablet c87cdec4d1ca41098090cb9e6eda5ca2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:33:25 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:40406
I20260925 16:33:25.903980 7882 raft_consensus.cc:2229] T c87cdec4d1ca41098090cb9e6eda5ca2 P 8d93ff6ea52e43f295aa3c9846e05303 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:33:25.904106 7882 raft_consensus.cc:2258] T c87cdec4d1ca41098090cb9e6eda5ca2 P 8d93ff6ea52e43f295aa3c9846e05303 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:33:25.904100 7884 tablet_replica.cc:330] T c87cdec4d1ca41098090cb9e6eda5ca2 P 0e11f3b8048a40d093ebd83dfdfe0b46: stopping tablet replica
I20260925 16:33:25.904235 7884 raft_consensus.cc:2229] T c87cdec4d1ca41098090cb9e6eda5ca2 P 0e11f3b8048a40d093ebd83dfdfe0b46 [term 1 LEADER]: Raft consensus shutting down.
I20260925 16:33:25.904335 7883 tablet_replica.cc:330] T c87cdec4d1ca41098090cb9e6eda5ca2 P 37e3504cf5f848cc912031815590b8a8: stopping tablet replica
I20260925 16:33:25.904350 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 7264
I20260925 16:33:25.904912 7884 raft_consensus.cc:2258] T c87cdec4d1ca41098090cb9e6eda5ca2 P 0e11f3b8048a40d093ebd83dfdfe0b46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:33:25.905110 7882 ts_tablet_manager.cc:1905] T c87cdec4d1ca41098090cb9e6eda5ca2 P 8d93ff6ea52e43f295aa3c9846e05303: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:33:25.906817 7884 ts_tablet_manager.cc:1905] T c87cdec4d1ca41098090cb9e6eda5ca2 P 0e11f3b8048a40d093ebd83dfdfe0b46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:33:25.907532 7882 ts_tablet_manager.cc:1918] T c87cdec4d1ca41098090cb9e6eda5ca2 P 8d93ff6ea52e43f295aa3c9846e05303: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:33:25.907661 7882 log.cc:1196] T c87cdec4d1ca41098090cb9e6eda5ca2 P 8d93ff6ea52e43f295aa3c9846e05303: Deleting WAL directory at /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-2/wal/wals/c87cdec4d1ca41098090cb9e6eda5ca2
I20260925 16:33:25.907994 7882 ts_tablet_manager.cc:1939] T c87cdec4d1ca41098090cb9e6eda5ca2 P 8d93ff6ea52e43f295aa3c9846e05303: Deleting consensus metadata
I20260925 16:33:25.908777 7106 catalog_manager.cc:4841] TS 8d93ff6ea52e43f295aa3c9846e05303 (127.0.98.3:42375): tablet c87cdec4d1ca41098090cb9e6eda5ca2 (table test-table [id=9c2aaeeda98546bd91e186ac257f7a15]) successfully deleted
I20260925 16:33:25.909214 7884 ts_tablet_manager.cc:1918] T c87cdec4d1ca41098090cb9e6eda5ca2 P 0e11f3b8048a40d093ebd83dfdfe0b46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:33:25.909274 7884 log.cc:1196] T c87cdec4d1ca41098090cb9e6eda5ca2 P 0e11f3b8048a40d093ebd83dfdfe0b46: Deleting WAL directory at /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353966814071-392-0/minicluster-data/ts-1/wal/wals/c87cdec4d1ca41098090cb9e6eda5ca2
I20260925 16:33:25.909478 7884 ts_tablet_manager.cc:1939] T c87cdec4d1ca41098090cb9e6eda5ca2 P 0e11f3b8048a40d093ebd83dfdfe0b46: Deleting consensus metadata
I20260925 16:33:25.910079 7105 catalog_manager.cc:4841] TS 0e11f3b8048a40d093ebd83dfdfe0b46 (127.0.98.2:43771): tablet c87cdec4d1ca41098090cb9e6eda5ca2 (table test-table [id=9c2aaeeda98546bd91e186ac257f7a15]) successfully deleted
I20260925 16:33:25.912899 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 7492
I20260925 16:33:25.913039 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:25.912755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43207 (local address 127.0.0.1:43240)
0925 16:33:25.912877 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:25.912932 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 16:33:25.913797 7107 catalog_manager.cc:4584] TS 37e3504cf5f848cc912031815590b8a8 (127.0.98.1:43207): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c87cdec4d1ca41098090cb9e6eda5ca2: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43207: connect: Connection refused (error 111)
I20260925 16:33:25.919610 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 7628
I20260925 16:33:25.925115 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 6984
I20260925 16:33:25.931352 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 7092
2026-09-25T16:33:25Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7012 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:33:25 dist-test-slave-65pf krb5kdc[7887](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:33:25 dist-test-slave-65pf krb5kdc[7887](info): set up 2 sockets
Sep 25 16:33:25 dist-test-slave-65pf krb5kdc[7887](info): commencing operation
krb5kdc: starting...
W20260925 16:33:26.967087 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/krb5kdc/test-user.keytab.
Sep 25 16:33:27 dist-test-slave-65pf krb5kdc[7887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:33:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:33:27Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260925 16:33:27.085068 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37535
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37535
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:33:27.191946 7900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:33:27.192184 7900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:33:27.192221 7900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:33:27.196115 7900 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:33:27.196162 7900 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 16:33:27.196178 7900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:33:27.196197 7900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:33:27.196250 7900 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:33:27.196281 7900 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:33:27.201299 7900 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37535
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:33:27.202286 7900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:33:27.203299 7900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:33:27.211310 7906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:27.211333 7905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:27.211280 7908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:27.211609 7900 server_base.cc:1034] running on GCE node
I20260925 16:33:27.212102 7900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:33:27.213150 7900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:33:27.214344 7900 hybrid_clock.cc:648] HybridClock initialized: now 1790354007214327 us; error 30 us; skew 500 ppm
Sep 25 16:33:27 dist-test-slave-65pf krb5kdc[7887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354007, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:33:27.217005 7900 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260925 16:33:27.217872 7900 webserver.cc:460] Webserver started at http://127.0.98.62:46149/ using document root <none> and password file <none>
I20260925 16:33:27.218451 7900 fs_manager.cc:359] Metadata directory not provided
I20260925 16:33:27.218542 7900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:33:27.218787 7900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:33:27.220831 7900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/master-0/data/instance:
uuid: "7bf7b1c2fbb04b7a96e4038edbe73e92"
format_stamp: "Formatted at 2026-09-25 16:33:27 on dist-test-slave-65pf"
server_key: "2571a319ebaf8108be798d1a1ecb71a1"
server_key_iv: "449ce558a109dd70de43025427bf1c07"
server_key_version: "encryptionkey@0"
I20260925 16:33:27.221410 7900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/master-0/wal/instance:
uuid: "7bf7b1c2fbb04b7a96e4038edbe73e92"
format_stamp: "Formatted at 2026-09-25 16:33:27 on dist-test-slave-65pf"
server_key: "2571a319ebaf8108be798d1a1ecb71a1"
server_key_iv: "449ce558a109dd70de43025427bf1c07"
server_key_version: "encryptionkey@0"
I20260925 16:33:27.225066 7900 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260925 16:33:27.227473 7915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:27.228646 7900 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:33:27.228788 7900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/master-0/wal
uuid: "7bf7b1c2fbb04b7a96e4038edbe73e92"
format_stamp: "Formatted at 2026-09-25 16:33:27 on dist-test-slave-65pf"
server_key: "2571a319ebaf8108be798d1a1ecb71a1"
server_key_iv: "449ce558a109dd70de43025427bf1c07"
server_key_version: "encryptionkey@0"
I20260925 16:33:27.228894 7900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:33:27.259783 7900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:33:27.262827 7900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:33:27.263036 7900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:33:27.271013 7900 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37535
I20260925 16:33:27.271039 7977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37535 every 8 connection(s)
I20260925 16:33:27.272024 7900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:33:27.274698 7978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:33:27.280058 7978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bf7b1c2fbb04b7a96e4038edbe73e92: Bootstrap starting.
I20260925 16:33:27.280956 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 7900
I20260925 16:33:27.281100 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/master-0/wal/instance
I20260925 16:33:27.281392 392 external_mini_cluster.cc:1428] Setting key 0f5b8933c185ab229453a73034e15b8b
I20260925 16:33:27.282914 7978 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7bf7b1c2fbb04b7a96e4038edbe73e92: Neither blocks nor log segments found. Creating new log.
I20260925 16:33:27.283772 7978 log.cc:824] T 00000000000000000000000000000000 P 7bf7b1c2fbb04b7a96e4038edbe73e92: Log is configured to *not* fsync() on all Append() calls
I20260925 16:33:27.285936 7978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bf7b1c2fbb04b7a96e4038edbe73e92: No bootstrap required, opened a new log
I20260925 16:33:27.290066 7978 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7bf7b1c2fbb04b7a96e4038edbe73e92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bf7b1c2fbb04b7a96e4038edbe73e92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37535 } }
I20260925 16:33:27.290325 7978 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7bf7b1c2fbb04b7a96e4038edbe73e92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
Sep 25 16:33:27 dist-test-slave-65pf krb5kdc[7887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 16:33:27.290421 7978 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7bf7b1c2fbb04b7a96e4038edbe73e92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bf7b1c2fbb04b7a96e4038edbe73e92, State: Initialized, Role: FOLLOWER
I20260925 16:33:27.290978 7978 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7bf7b1c2fbb04b7a96e4038edbe73e92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bf7b1c2fbb04b7a96e4038edbe73e92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37535 } }
I20260925 16:33:27.291177 7978 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7bf7b1c2fbb04b7a96e4038edbe73e92 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:33:27.291270 7978 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7bf7b1c2fbb04b7a96e4038edbe73e92 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:33:27.291368 7978 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7bf7b1c2fbb04b7a96e4038edbe73e92 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:33:27.292470 7978 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7bf7b1c2fbb04b7a96e4038edbe73e92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bf7b1c2fbb04b7a96e4038edbe73e92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37535 } }
I20260925 16:33:27.292810 7978 leader_election.cc:302] T 00000000000000000000000000000000 P 7bf7b1c2fbb04b7a96e4038edbe73e92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7bf7b1c2fbb04b7a96e4038edbe73e92; no voters:
I20260925 16:33:27.293154 7978 leader_election.cc:288] T 00000000000000000000000000000000 P 7bf7b1c2fbb04b7a96e4038edbe73e92 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:33:27.293280 7983 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7bf7b1c2fbb04b7a96e4038edbe73e92 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:33:27.293505 7983 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7bf7b1c2fbb04b7a96e4038edbe73e92 [term 1 LEADER]: Becoming Leader. State: Replica: 7bf7b1c2fbb04b7a96e4038edbe73e92, State: Running, Role: LEADER
I20260925 16:33:27.293860 7983 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7bf7b1c2fbb04b7a96e4038edbe73e92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bf7b1c2fbb04b7a96e4038edbe73e92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37535 } }
I20260925 16:33:27.294520 7978 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7bf7b1c2fbb04b7a96e4038edbe73e92 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:33:27.296006 7984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bf7b1c2fbb04b7a96e4038edbe73e92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7bf7b1c2fbb04b7a96e4038edbe73e92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bf7b1c2fbb04b7a96e4038edbe73e92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37535 } } }
I20260925 16:33:27.296131 7984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bf7b1c2fbb04b7a96e4038edbe73e92 [sys.catalog]: This master's current role is: LEADER
I20260925 16:33:27.296445 7985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bf7b1c2fbb04b7a96e4038edbe73e92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7bf7b1c2fbb04b7a96e4038edbe73e92. Latest consensus state: current_term: 1 leader_uuid: "7bf7b1c2fbb04b7a96e4038edbe73e92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bf7b1c2fbb04b7a96e4038edbe73e92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37535 } } }
I20260925 16:33:27.296641 7985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bf7b1c2fbb04b7a96e4038edbe73e92 [sys.catalog]: This master's current role is: LEADER
I20260925 16:33:27.296725 7981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:27.282523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36606 (local address 127.0.98.62:37535)
0925 16:33:27.283041 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:27.283068 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:27.283115 (+ 47us) server_negotiation.cc:407] Connection header received
0925 16:33:27.283916 (+ 801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:27.283938 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:27.284335 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:27.284702 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:27.285352 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:27.286831 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:27.287674 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:27.288432 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:27.288602 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:27.291092 (+ 2490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:27.291097 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:27.291109 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:27.291134 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:27.293018 (+ 1884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:27.293536 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:27.293542 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:27.293548 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:27.293685 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:27.294054 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:27.294057 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:27.294058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:27.294561 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:27.294773 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:27.295212 (+ 439us) server_negotiation.cc:299] Negotiation successful
0925 16:33:27.295550 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":146,"threads_started":1}
W20260925 16:33:27.299912 7998 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7bf7b1c2fbb04b7a96e4038edbe73e92: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 16:33:27.299978 7998 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 16:33:27.300060 7992 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:33:27.300818 7992 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:33:27.306216 7992 catalog_manager.cc:1347] Generated new cluster ID: 6cf621e1853c4f88b5d79405825b329a
I20260925 16:33:27.306280 7992 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:33:27.324775 7992 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:33:27.325841 7992 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:33:27.339906 7992 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7bf7b1c2fbb04b7a96e4038edbe73e92: Generated new TSK 0
I20260925 16:33:27.340643 7992 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260925 16:33:27.377622 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37535
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:33:27.502053 8004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:33:27.502302 8004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:33:27.502343 8004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:33:27.506367 8004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:33:27.506428 8004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:33:27.506465 8004 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 16:33:27.506497 8004 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 16:33:27.511632 8004 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:37535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:33:27.512609 8004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:33:27.513664 8004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:33:27.522936 8010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:27.522990 8012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:27.523118 8009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:27.523135 8004 server_base.cc:1034] running on GCE node
I20260925 16:33:27.523665 8004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:33:27.524310 8004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:33:27.525502 8004 hybrid_clock.cc:648] HybridClock initialized: now 1790354007525494 us; error 29 us; skew 500 ppm
Sep 25 16:33:27 dist-test-slave-65pf krb5kdc[7887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354007, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:33:27.528290 8004 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260925 16:33:27.529150 8004 webserver.cc:460] Webserver started at http://127.0.98.1:36545/ using document root <none> and password file <none>
I20260925 16:33:27.529969 8004 fs_manager.cc:359] Metadata directory not provided
I20260925 16:33:27.530054 8004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:33:27.530287 8004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:33:27.532447 8004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/ts-0/data/instance:
uuid: "948c869ed1fe452e8f64517c105e455d"
format_stamp: "Formatted at 2026-09-25 16:33:27 on dist-test-slave-65pf"
server_key: "05821a3640460b7bce74165992c1da76"
server_key_iv: "8bee4d6d51be8867023d8747fe97d240"
server_key_version: "encryptionkey@0"
I20260925 16:33:27.533049 8004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/ts-0/wal/instance:
uuid: "948c869ed1fe452e8f64517c105e455d"
format_stamp: "Formatted at 2026-09-25 16:33:27 on dist-test-slave-65pf"
server_key: "05821a3640460b7bce74165992c1da76"
server_key_iv: "8bee4d6d51be8867023d8747fe97d240"
server_key_version: "encryptionkey@0"
I20260925 16:33:27.536713 8004 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 16:33:27.539072 8019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:27.540199 8004 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:33:27.540340 8004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/ts-0/wal
uuid: "948c869ed1fe452e8f64517c105e455d"
format_stamp: "Formatted at 2026-09-25 16:33:27 on dist-test-slave-65pf"
server_key: "05821a3640460b7bce74165992c1da76"
server_key_iv: "8bee4d6d51be8867023d8747fe97d240"
server_key_version: "encryptionkey@0"
I20260925 16:33:27.540449 8004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:33:27.554162 8004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:33:27.557439 8004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:33:27.557643 8004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:33:27.559202 8004 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:33:27.559274 8004 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:27.559341 8004 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:33:27.559394 8004 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 16:33:27 dist-test-slave-65pf krb5kdc[7887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354007, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 16:33:27.570498 8004 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41731
I20260925 16:33:27.570606 8134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41731 every 8 connection(s)
I20260925 16:33:27.571643 8004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
I20260925 16:33:27.573568 7981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:27.559853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37535 (local address 127.0.98.62:37535)
0925 16:33:27.559994 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:27.559999 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:27.560930 (+ 931us) server_negotiation.cc:407] Connection header received
0925 16:33:27.562207 (+ 1277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:27.562212 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:27.562295 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:27.562377 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:27.564248 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:27.565105 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:27.566609 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:27.567075 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:27.567228 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:27.569762 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:27.569766 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:27.569769 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:27.569795 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:27.571526 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:27.572098 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:27.572103 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:27.572106 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:27.572191 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:27.572624 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:27.572628 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:27.572629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:27.572820 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:27.572903 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:27.573334 (+ 431us) server_negotiation.cc:299] Negotiation successful
0925 16:33:27.573448 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 16:33:27.574430 8028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:27.560273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37535 (local address 127.0.98.1:37535)
0925 16:33:27.560745 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:27.560802 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:27.561916 (+ 1114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:27.562858 (+ 942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:27.562868 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:27.563383 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:27.564079 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:27.564103 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:27.565225 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:27.565228 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:27.566446 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:27.566457 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:27.567299 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:27.567307 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:27.567479 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:27.568076 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:27.568091 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:27.569593 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:33:27.571644 (+ 2051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:27.571649 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:27.571660 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:27.571986 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:27.572320 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:27.572323 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:27.572325 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:27.572517 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:27.572913 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:33:27.572918 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:33:27.573134 (+ 216us) client_negotiation.cc:769] Sending connection context
0925 16:33:27.573285 (+ 151us) client_negotiation.cc:240] Negotiation successful
0925 16:33:27.573533 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":103,"threads_started":1}
I20260925 16:33:27.574499 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 8004
I20260925 16:33:27.574635 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/ts-0/wal/instance
I20260925 16:33:27.575017 392 external_mini_cluster.cc:1428] Setting key 2fa8301c6a6c2151e45e3c73b8ebf05c
I20260925 16:33:27.576013 8135 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37535
I20260925 16:33:27.576160 8135 heartbeater.cc:464] Registering TS with master...
I20260925 16:33:27.576746 8135 heartbeater.cc:511] Master 127.0.98.62:37535 requested a full tablet report, sending...
I20260925 16:33:27.578446 7931 ts_manager.cc:194] Registered new tserver with Master: 948c869ed1fe452e8f64517c105e455d (127.0.98.1:41731)
I20260925 16:33:27.579859 7931 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37535
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 16:33:27.586400 7981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:27.578231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50081 (local address 127.0.98.62:37535)
0925 16:33:27.578376 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:27.578383 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:27.578758 (+ 375us) server_negotiation.cc:407] Connection header received
0925 16:33:27.578870 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:27.578873 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:27.578913 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:27.579003 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:27.579561 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:27.580259 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:27.581614 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:27.581932 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:27.582039 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:27.583436 (+ 1397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:27.583439 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:27.583441 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:27.583462 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:27.584858 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:27.585338 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:27.585341 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:27.585343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:27.585435 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:27.585731 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:27.585733 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:27.585735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:27.585890 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:27.585975 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:27.586198 (+ 223us) server_negotiation.cc:299] Negotiation successful
0925 16:33:27.586291 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 16:33:27.586477 8142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:27.578340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37535 (local address 127.0.98.1:50081)
0925 16:33:27.578658 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:27.578675 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:27.578780 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:27.579039 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:27.579042 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:27.579277 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:27.579430 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:27.579440 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:27.580397 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:27.580404 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:27.581479 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:27.581493 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:27.582067 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:27.582073 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:27.582204 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:27.582811 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:27.582825 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:27.583288 (+ 463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:33:27.584973 (+ 1685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:27.584980 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:27.584984 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:27.585218 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:27.585526 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:27.585529 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:27.585532 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:27.585638 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:27.585983 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:33:27.585985 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:33:27.586061 (+ 76us) client_negotiation.cc:769] Sending connection context
0925 16:33:27.586144 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 16:33:27.586250 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":100,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260925 16:33:27.603641 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37535
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:33:27.713032 8144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:33:27.713305 8144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:33:27.713348 8144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:33:27.717485 8144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:33:27.717547 8144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:33:27.717613 8144 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 16:33:27.717679 8144 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 16:33:27.722890 8144 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:37535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:33:27.723989 8144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:33:27.725035 8144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:33:27.734750 8149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:27.734725 8144 server_base.cc:1034] running on GCE node
W20260925 16:33:27.734709 8150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:27.734931 8152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:27.735306 8144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:33:27.735914 8144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:33:27.737103 8144 hybrid_clock.cc:648] HybridClock initialized: now 1790354007737066 us; error 53 us; skew 500 ppm
Sep 25 16:33:27 dist-test-slave-65pf krb5kdc[7887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354007, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:33:27.739828 8144 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260925 16:33:27.740676 8144 webserver.cc:460] Webserver started at http://127.0.98.2:34141/ using document root <none> and password file <none>
I20260925 16:33:27.741185 8144 fs_manager.cc:359] Metadata directory not provided
I20260925 16:33:27.741242 8144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:33:27.741402 8144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:33:27.743432 8144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/ts-1/data/instance:
uuid: "f649366115124f25bc74e77e773613ad"
format_stamp: "Formatted at 2026-09-25 16:33:27 on dist-test-slave-65pf"
server_key: "c42e0e224341bc336f27bcf758351ae1"
server_key_iv: "762102de034da6324b3f23c8daa18f9a"
server_key_version: "encryptionkey@0"
I20260925 16:33:27.743939 8144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f649366115124f25bc74e77e773613ad"
format_stamp: "Formatted at 2026-09-25 16:33:27 on dist-test-slave-65pf"
server_key: "c42e0e224341bc336f27bcf758351ae1"
server_key_iv: "762102de034da6324b3f23c8daa18f9a"
server_key_version: "encryptionkey@0"
I20260925 16:33:27.747611 8144 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 16:33:27.749910 8159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:27.751151 8144 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:33:27.751243 8144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/ts-1/wal
uuid: "f649366115124f25bc74e77e773613ad"
format_stamp: "Formatted at 2026-09-25 16:33:27 on dist-test-slave-65pf"
server_key: "c42e0e224341bc336f27bcf758351ae1"
server_key_iv: "762102de034da6324b3f23c8daa18f9a"
server_key_version: "encryptionkey@0"
I20260925 16:33:27.751328 8144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:33:27.765167 8144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:33:27.768190 8144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:33:27.768343 8144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:33:27.769768 8144 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:33:27.769870 8144 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:27.769989 8144 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:33:27.770042 8144 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 16:33:27 dist-test-slave-65pf krb5kdc[7887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354007, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 16:33:27.780814 8144 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40387
I20260925 16:33:27.781144 8274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40387 every 8 connection(s)
I20260925 16:33:27.781745 8144 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
I20260925 16:33:27.783761 7981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:27.770692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37923 (local address 127.0.98.62:37535)
0925 16:33:27.770830 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:27.770833 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:27.771880 (+ 1047us) server_negotiation.cc:407] Connection header received
0925 16:33:27.772803 (+ 923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:27.772806 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:27.772864 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:27.772945 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:27.774219 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:27.774932 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:27.776346 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:27.776636 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:27.776770 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:27.779948 (+ 3178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:27.779952 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:27.779954 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:27.779982 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:27.781611 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:27.782219 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:27.782222 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:27.782224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:27.782308 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:27.782693 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:27.782697 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:27.782699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:27.782886 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:27.782994 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:27.783474 (+ 480us) server_negotiation.cc:299] Negotiation successful
0925 16:33:27.783593 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 16:33:27.784498 8168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:27.771227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37535 (local address 127.0.98.2:37923)
0925 16:33:27.771714 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:27.771767 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:27.772595 (+ 828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:27.773299 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:27.773306 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:27.773631 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:27.774075 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:27.774087 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:27.775056 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:27.775062 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:27.776206 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:27.776218 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:27.776818 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:27.776823 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:27.777021 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:27.777716 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:27.777735 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:27.779773 (+ 2038us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:33:27.781731 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:27.781745 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:27.781759 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:27.782097 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:27.782410 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:27.782413 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:27.782415 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:27.782570 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:27.782992 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:33:27.782999 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:33:27.783234 (+ 235us) client_negotiation.cc:769] Sending connection context
0925 16:33:27.783415 (+ 181us) client_negotiation.cc:240] Negotiation successful
0925 16:33:27.783719 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":146,"threads_started":1}
I20260925 16:33:27.785625 8275 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37535
I20260925 16:33:27.785753 8275 heartbeater.cc:464] Registering TS with master...
I20260925 16:33:27.786286 8275 heartbeater.cc:511] Master 127.0.98.62:37535 requested a full tablet report, sending...
I20260925 16:33:27.787364 7931 ts_manager.cc:194] Registered new tserver with Master: f649366115124f25bc74e77e773613ad (127.0.98.2:40387)
I20260925 16:33:27.788128 7931 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37923
I20260925 16:33:27.789338 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 8144
I20260925 16:33:27.789481 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/ts-1/wal/instance
I20260925 16:33:27.789783 392 external_mini_cluster.cc:1428] Setting key ee042408696b9619450d96dd721f30cb
I20260925 16:33:27.796137 8281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:27.787908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37535 (local address 127.0.98.2:56335)
0925 16:33:27.788236 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:27.788269 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:27.788364 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:27.788617 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:27.788619 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:27.788836 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:27.788946 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:27.788971 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:27.789941 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:27.789944 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:27.790745 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:27.790753 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:27.791298 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:27.791301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:27.791390 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:27.791879 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:27.791900 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:27.792377 (+ 477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:33:27.794379 (+ 2002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:27.794382 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:27.794383 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:27.794631 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:27.794950 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:27.794952 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:27.794953 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:27.795030 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:27.795552 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:33:27.795554 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:33:27.795628 (+ 74us) client_negotiation.cc:769] Sending connection context
0925 16:33:27.795725 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 16:33:27.795928 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":87,"threads_started":1}
I20260925 16:33:27.796183 7981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:27.787993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56335 (local address 127.0.98.62:37535)
0925 16:33:27.788104 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:27.788107 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:27.788325 (+ 218us) server_negotiation.cc:407] Connection header received
0925 16:33:27.788463 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:27.788466 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:27.788507 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:27.788593 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:27.789095 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:27.789827 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:27.790859 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:27.791184 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:27.791267 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:27.792515 (+ 1248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:27.792518 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:27.792520 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:27.792540 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:27.794264 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:27.794716 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:27.794723 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:27.794728 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:27.794818 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:27.795150 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:27.795156 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:27.795161 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:27.795448 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:27.795557 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:27.795812 (+ 255us) server_negotiation.cc:299] Negotiation successful
0925 16:33:27.795969 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260925 16:33:27.817802 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37535
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:33:27.929720 8284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:33:27.929972 8284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:33:27.930048 8284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:33:27.934442 8284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:33:27.934502 8284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:33:27.934569 8284 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 16:33:27.934664 8284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 16:33:27.939843 8284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:37535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:33:27.940881 8284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:33:27.941908 8284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:33:27.951637 8292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:27.951632 8289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:27.951632 8290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:27.951925 8284 server_base.cc:1034] running on GCE node
I20260925 16:33:27.952410 8284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:33:27.953070 8284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:33:27.954275 8284 hybrid_clock.cc:648] HybridClock initialized: now 1790354007954252 us; error 37 us; skew 500 ppm
Sep 25 16:33:27 dist-test-slave-65pf krb5kdc[7887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354007, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:33:27.957254 8284 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260925 16:33:27.958141 8284 webserver.cc:460] Webserver started at http://127.0.98.3:37557/ using document root <none> and password file <none>
I20260925 16:33:27.958762 8284 fs_manager.cc:359] Metadata directory not provided
I20260925 16:33:27.958854 8284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:33:27.959081 8284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:33:27.961122 8284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/ts-2/data/instance:
uuid: "9203d42b2ff045c386ade97cea8d1430"
format_stamp: "Formatted at 2026-09-25 16:33:27 on dist-test-slave-65pf"
server_key: "a2f9c57d44cffec0417e987fa6176ada"
server_key_iv: "3e134203ea39e17bf8f72ddc7dd4e784"
server_key_version: "encryptionkey@0"
I20260925 16:33:27.961663 8284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9203d42b2ff045c386ade97cea8d1430"
format_stamp: "Formatted at 2026-09-25 16:33:27 on dist-test-slave-65pf"
server_key: "a2f9c57d44cffec0417e987fa6176ada"
server_key_iv: "3e134203ea39e17bf8f72ddc7dd4e784"
server_key_version: "encryptionkey@0"
I20260925 16:33:27.965417 8284 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260925 16:33:27.967815 8299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:27.968966 8284 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 16:33:27.969106 8284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/ts-2/wal
uuid: "9203d42b2ff045c386ade97cea8d1430"
format_stamp: "Formatted at 2026-09-25 16:33:27 on dist-test-slave-65pf"
server_key: "a2f9c57d44cffec0417e987fa6176ada"
server_key_iv: "3e134203ea39e17bf8f72ddc7dd4e784"
server_key_version: "encryptionkey@0"
I20260925 16:33:27.969215 8284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:33:27.990967 8284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:33:27.994300 8284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:33:27.994516 8284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:33:27.996194 8284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:33:27.996282 8284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:27.996367 8284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:33:27.996405 8284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 16:33:28 dist-test-slave-65pf krb5kdc[7887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354007, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 16:33:28.007283 8284 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36421
I20260925 16:33:28.007437 8414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36421 every 8 connection(s)
I20260925 16:33:28.008316 8284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
I20260925 16:33:28.010040 7981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:27.996955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56655 (local address 127.0.98.62:37535)
0925 16:33:27.997071 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:27.997075 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:27.998087 (+ 1012us) server_negotiation.cc:407] Connection header received
0925 16:33:27.999143 (+ 1056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:27.999146 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:27.999201 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:27.999308 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:28.000546 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:28.001199 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:28.002711 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:28.002942 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:28.003061 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:28.006121 (+ 3060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:28.006127 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:28.006132 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:28.006159 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:28.007900 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:28.008503 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:28.008509 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:28.008512 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:28.008596 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:28.008966 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:28.008969 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:28.008971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:28.009202 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:28.009281 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:28.009736 (+ 455us) server_negotiation.cc:299] Negotiation successful
0925 16:33:28.009869 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 16:33:28.011147 8308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:27.997476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37535 (local address 127.0.98.3:56655)
0925 16:33:27.997946 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:27.997987 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:27.998943 (+ 956us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:27.999477 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:27.999487 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:27.999880 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:28.000408 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:28.000423 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:28.001323 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:28.001330 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:28.002567 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:28.002617 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:28.003106 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:28.003110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:28.003324 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:28.003960 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:28.003984 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:28.005925 (+ 1941us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:33:28.008011 (+ 2086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:28.008017 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:28.008030 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:28.008354 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:28.008681 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:28.008685 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:28.008687 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:28.008852 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:28.009314 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:33:28.009319 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:33:28.009531 (+ 212us) client_negotiation.cc:769] Sending connection context
0925 16:33:28.009687 (+ 156us) client_negotiation.cc:240] Negotiation successful
0925 16:33:28.010058 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":132,"threads_started":1}
I20260925 16:33:28.012174 8415 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37535
I20260925 16:33:28.012328 8415 heartbeater.cc:464] Registering TS with master...
I20260925 16:33:28.012814 8415 heartbeater.cc:511] Master 127.0.98.62:37535 requested a full tablet report, sending...
I20260925 16:33:28.013823 7932 ts_manager.cc:194] Registered new tserver with Master: 9203d42b2ff045c386ade97cea8d1430 (127.0.98.3:36421)
I20260925 16:33:28.014288 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 8284
I20260925 16:33:28.014410 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/ts-2/wal/instance
I20260925 16:33:28.014653 7932 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56655
I20260925 16:33:28.014781 392 external_mini_cluster.cc:1428] Setting key 88d3ef576ee5d4ea6b54b2558c3d40f0
I20260925 16:33:28.016837 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 16:33:28.021901 8421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:28.014332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37535 (local address 127.0.98.3:49835)
0925 16:33:28.014786 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:28.014801 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:28.014932 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:28.015195 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:28.015198 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:28.015408 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:28.015581 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:28.015590 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:28.016442 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:28.016445 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:28.017126 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:28.017135 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:28.017582 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:28.017585 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:28.017675 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:28.018175 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:28.018186 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:28.018541 (+ 355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:33:28.020342 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:28.020348 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:28.020350 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:28.020615 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:28.020913 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:28.020919 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:28.020922 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:28.021017 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:28.021359 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:33:28.021362 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:33:28.021451 (+ 89us) client_negotiation.cc:769] Sending connection context
0925 16:33:28.021584 (+ 133us) client_negotiation.cc:240] Negotiation successful
0925 16:33:28.021732 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"mutex_wait_us":2,"thread_start_us":101,"threads_started":1}
I20260925 16:33:28.021992 7981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:28.014432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49835 (local address 127.0.98.62:37535)
0925 16:33:28.014540 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:28.014544 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:28.014869 (+ 325us) server_negotiation.cc:407] Connection header received
0925 16:33:28.015044 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:28.015048 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:28.015095 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:28.015177 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:28.015679 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:28.016317 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:28.017242 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:28.017472 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:28.017581 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:28.018751 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:28.018754 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:28.018756 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:28.018776 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:28.020230 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:28.020728 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:28.020732 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:28.020734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:28.020819 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:28.021107 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:28.021109 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:28.021111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:28.021265 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:28.021334 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:28.021646 (+ 312us) server_negotiation.cc:299] Negotiation successful
0925 16:33:28.021778 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
Sep 25 16:33:28 dist-test-slave-65pf krb5kdc[7887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354008, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 16:33:28 dist-test-slave-65pf krb5kdc[7887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:33:28 dist-test-slave-65pf krb5kdc[7887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 16:33:28.040025 7981 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:33:28.034287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36610 (local address 127.0.98.62:37535)
0925 16:33:28.034405 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:28.034408 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:28.034422 (+ 14us) server_negotiation.cc:407] Connection header received
0925 16:33:28.034459 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:28.034461 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:28.034498 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:28.034570 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:28.035295 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:28.035798 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:28.036603 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:28.036814 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:28.036888 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:28.039906 (+ 3018us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36610: BlockingRecv error: recv got EOF from 127.0.0.1:36610 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 25 16:33:28 dist-test-slave-65pf krb5kdc[7887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354008, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 16:33:28.049577 7981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:28.041315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36614 (local address 127.0.98.62:37535)
0925 16:33:28.041452 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:28.041458 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:28.041588 (+ 130us) server_negotiation.cc:407] Connection header received
0925 16:33:28.041730 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:28.041733 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:28.041777 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:28.041859 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:28.042420 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:28.042970 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:28.043751 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:28.044052 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:28.044157 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:28.046445 (+ 2288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:28.046449 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:28.046452 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:28.046486 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:28.047962 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:28.048378 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:28.048384 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:28.048385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:28.048474 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:28.048751 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:28.048754 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:28.048756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:28.048938 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:28.049076 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:28.049301 (+ 225us) server_negotiation.cc:299] Negotiation successful
0925 16:33:28.049409 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 16:33:28.052496 7932 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36614:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 16:33:28.055682 7932 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:33:28.069998 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:28.064557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58224 (local address 127.0.98.3:36421)
0925 16:33:28.065365 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:28.065373 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:28.065393 (+ 20us) server_negotiation.cc:407] Connection header received
0925 16:33:28.065453 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:28.065457 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:28.065609 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:28.065729 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:33:28.065991 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:28.067156 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:28.068634 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:28.069333 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:28.069473 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:28.069642 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:28.069756 (+ 114us) server_negotiation.cc:299] Negotiation successful
0925 16:33:28.069844 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":728,"thread_start_us":105,"threads_started":1}
I20260925 16:33:28.069988 8440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:28.063936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36421 (local address 127.0.0.1:58224)
0925 16:33:28.064496 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:28.064694 (+ 198us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:28.064775 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:28.065743 (+ 968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:28.065748 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:28.065767 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:33:28.065865 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:28.065870 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:28.067262 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:28.067267 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:28.068515 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:28.068524 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:28.069473 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:28.069476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:28.069577 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:28.069610 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 16:33:28.069728 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 16:33:28.069844 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"mutex_wait_us":32,"thread_start_us":87,"threads_started":1}
I20260925 16:33:28.070654 8439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:28.064149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40387 (local address 127.0.0.1:58520)
0925 16:33:28.065908 (+ 1759us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:28.065945 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:28.066044 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:28.066418 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:28.066421 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:28.066438 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:33:28.066530 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:28.066539 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:28.067822 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:28.067825 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:28.068951 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:28.068960 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:28.070239 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:28.070243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:28.070339 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:28.070364 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 16:33:28.070430 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 16:33:28.070497 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1710,"mutex_wait_us":10,"thread_start_us":99,"threads_started":1}
I20260925 16:33:28.071290 8442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:28.064930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58520 (local address 127.0.98.2:40387)
0925 16:33:28.065262 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:28.065272 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:28.066047 (+ 775us) server_negotiation.cc:407] Connection header received
0925 16:33:28.066146 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:28.066150 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:28.066286 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:28.066434 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:33:28.066688 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:28.067680 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:28.069080 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:28.069882 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:28.069982 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:28.070995 (+ 1013us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:28.071049 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 16:33:28.071146 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":115,"threads_started":1}
I20260925 16:33:28.071523 8437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:28.063747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41731 (local address 127.0.0.1:51892)
0925 16:33:28.064497 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:28.064552 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:28.064682 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:28.066835 (+ 2153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:28.066838 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:28.066863 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:33:28.066940 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:28.066945 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:28.068454 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:28.068458 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:28.069424 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:28.069434 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:28.071145 (+ 1711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:28.071148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:28.071236 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:28.071256 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 16:33:28.071321 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 16:33:28.071407 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"mutex_wait_us":19,"thread_start_us":72,"threads_started":1}
I20260925 16:33:28.072163 8438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:28.063930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51892 (local address 127.0.98.1:41731)
0925 16:33:28.066269 (+ 2339us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:28.066284 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:28.066306 (+ 22us) server_negotiation.cc:407] Connection header received
0925 16:33:28.066375 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:28.066379 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:28.066540 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:28.066700 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:33:28.067085 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:28.068331 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:28.069565 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:28.070727 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:28.071738 (+ 1011us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:28.071896 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:28.071946 (+ 50us) server_negotiation.cc:299] Negotiation successful
0925 16:33:28.072036 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2242,"thread_start_us":90,"threads_started":1}
I20260925 16:33:28.073261 8349 tablet_service.cc:1467] Processing CreateTablet for tablet bec074d69bcd424392257622110dc61e (DEFAULT_TABLE table=test-table [id=dfbe188c5ec0421ba7b42537fbae9bd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:33:28.073860 8209 tablet_service.cc:1467] Processing CreateTablet for tablet bec074d69bcd424392257622110dc61e (DEFAULT_TABLE table=test-table [id=dfbe188c5ec0421ba7b42537fbae9bd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:33:28.074429 8349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bec074d69bcd424392257622110dc61e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:33:28.074903 8209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bec074d69bcd424392257622110dc61e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:33:28.077224 8069 tablet_service.cc:1467] Processing CreateTablet for tablet bec074d69bcd424392257622110dc61e (DEFAULT_TABLE table=test-table [id=dfbe188c5ec0421ba7b42537fbae9bd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:33:28.078017 8069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bec074d69bcd424392257622110dc61e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:33:28.082917 8443 tablet_bootstrap.cc:492] T bec074d69bcd424392257622110dc61e P 9203d42b2ff045c386ade97cea8d1430: Bootstrap starting.
I20260925 16:33:28.083365 8444 tablet_bootstrap.cc:492] T bec074d69bcd424392257622110dc61e P f649366115124f25bc74e77e773613ad: Bootstrap starting.
I20260925 16:33:28.084242 8445 tablet_bootstrap.cc:492] T bec074d69bcd424392257622110dc61e P 948c869ed1fe452e8f64517c105e455d: Bootstrap starting.
I20260925 16:33:28.085681 8444 tablet_bootstrap.cc:654] T bec074d69bcd424392257622110dc61e P f649366115124f25bc74e77e773613ad: Neither blocks nor log segments found. Creating new log.
I20260925 16:33:28.086476 8444 log.cc:824] T bec074d69bcd424392257622110dc61e P f649366115124f25bc74e77e773613ad: Log is configured to *not* fsync() on all Append() calls
I20260925 16:33:28.086719 8445 tablet_bootstrap.cc:654] T bec074d69bcd424392257622110dc61e P 948c869ed1fe452e8f64517c105e455d: Neither blocks nor log segments found. Creating new log.
I20260925 16:33:28.087435 8445 log.cc:824] T bec074d69bcd424392257622110dc61e P 948c869ed1fe452e8f64517c105e455d: Log is configured to *not* fsync() on all Append() calls
I20260925 16:33:28.088575 8444 tablet_bootstrap.cc:492] T bec074d69bcd424392257622110dc61e P f649366115124f25bc74e77e773613ad: No bootstrap required, opened a new log
I20260925 16:33:28.088723 8443 tablet_bootstrap.cc:654] T bec074d69bcd424392257622110dc61e P 9203d42b2ff045c386ade97cea8d1430: Neither blocks nor log segments found. Creating new log.
I20260925 16:33:28.088770 8444 ts_tablet_manager.cc:1397] T bec074d69bcd424392257622110dc61e P f649366115124f25bc74e77e773613ad: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 16:33:28.089334 8445 tablet_bootstrap.cc:492] T bec074d69bcd424392257622110dc61e P 948c869ed1fe452e8f64517c105e455d: No bootstrap required, opened a new log
I20260925 16:33:28.089438 8443 log.cc:824] T bec074d69bcd424392257622110dc61e P 9203d42b2ff045c386ade97cea8d1430: Log is configured to *not* fsync() on all Append() calls
I20260925 16:33:28.089524 8445 ts_tablet_manager.cc:1397] T bec074d69bcd424392257622110dc61e P 948c869ed1fe452e8f64517c105e455d: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260925 16:33:28.091070 8443 tablet_bootstrap.cc:492] T bec074d69bcd424392257622110dc61e P 9203d42b2ff045c386ade97cea8d1430: No bootstrap required, opened a new log
I20260925 16:33:28.091259 8443 ts_tablet_manager.cc:1397] T bec074d69bcd424392257622110dc61e P 9203d42b2ff045c386ade97cea8d1430: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260925 16:33:28.092725 8444 raft_consensus.cc:348] T bec074d69bcd424392257622110dc61e P f649366115124f25bc74e77e773613ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948c869ed1fe452e8f64517c105e455d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41731 } } peers { permanent_uuid: "9203d42b2ff045c386ade97cea8d1430" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36421 } } peers { permanent_uuid: "f649366115124f25bc74e77e773613ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } }
I20260925 16:33:28.093014 8444 raft_consensus.cc:374] T bec074d69bcd424392257622110dc61e P f649366115124f25bc74e77e773613ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:33:28.093075 8444 raft_consensus.cc:729] T bec074d69bcd424392257622110dc61e P f649366115124f25bc74e77e773613ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f649366115124f25bc74e77e773613ad, State: Initialized, Role: FOLLOWER
I20260925 16:33:28.093205 8445 raft_consensus.cc:348] T bec074d69bcd424392257622110dc61e P 948c869ed1fe452e8f64517c105e455d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948c869ed1fe452e8f64517c105e455d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41731 } } peers { permanent_uuid: "9203d42b2ff045c386ade97cea8d1430" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36421 } } peers { permanent_uuid: "f649366115124f25bc74e77e773613ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } }
I20260925 16:33:28.093442 8445 raft_consensus.cc:374] T bec074d69bcd424392257622110dc61e P 948c869ed1fe452e8f64517c105e455d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:33:28.093499 8445 raft_consensus.cc:729] T bec074d69bcd424392257622110dc61e P 948c869ed1fe452e8f64517c105e455d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948c869ed1fe452e8f64517c105e455d, State: Initialized, Role: FOLLOWER
I20260925 16:33:28.093532 8444 consensus_queue.cc:260] T bec074d69bcd424392257622110dc61e P f649366115124f25bc74e77e773613ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948c869ed1fe452e8f64517c105e455d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41731 } } peers { permanent_uuid: "9203d42b2ff045c386ade97cea8d1430" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36421 } } peers { permanent_uuid: "f649366115124f25bc74e77e773613ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } }
I20260925 16:33:28.093948 8445 consensus_queue.cc:260] T bec074d69bcd424392257622110dc61e P 948c869ed1fe452e8f64517c105e455d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948c869ed1fe452e8f64517c105e455d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41731 } } peers { permanent_uuid: "9203d42b2ff045c386ade97cea8d1430" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36421 } } peers { permanent_uuid: "f649366115124f25bc74e77e773613ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } }
I20260925 16:33:28.094218 8275 heartbeater.cc:503] Master 127.0.98.62:37535 was elected leader, sending a full tablet report...
I20260925 16:33:28.094558 8444 ts_tablet_manager.cc:1428] T bec074d69bcd424392257622110dc61e P f649366115124f25bc74e77e773613ad: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 16:33:28.094966 8445 ts_tablet_manager.cc:1428] T bec074d69bcd424392257622110dc61e P 948c869ed1fe452e8f64517c105e455d: Time spent starting tablet: real 0.005s user 0.004s sys 0.002s
I20260925 16:33:28.094893 8443 raft_consensus.cc:348] T bec074d69bcd424392257622110dc61e P 9203d42b2ff045c386ade97cea8d1430 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948c869ed1fe452e8f64517c105e455d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41731 } } peers { permanent_uuid: "9203d42b2ff045c386ade97cea8d1430" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36421 } } peers { permanent_uuid: "f649366115124f25bc74e77e773613ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } }
I20260925 16:33:28.095112 8443 raft_consensus.cc:374] T bec074d69bcd424392257622110dc61e P 9203d42b2ff045c386ade97cea8d1430 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:33:28.095160 8443 raft_consensus.cc:729] T bec074d69bcd424392257622110dc61e P 9203d42b2ff045c386ade97cea8d1430 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9203d42b2ff045c386ade97cea8d1430, State: Initialized, Role: FOLLOWER
I20260925 16:33:28.095351 8135 heartbeater.cc:503] Master 127.0.98.62:37535 was elected leader, sending a full tablet report...
I20260925 16:33:28.095554 8443 consensus_queue.cc:260] T bec074d69bcd424392257622110dc61e P 9203d42b2ff045c386ade97cea8d1430 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948c869ed1fe452e8f64517c105e455d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41731 } } peers { permanent_uuid: "9203d42b2ff045c386ade97cea8d1430" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36421 } } peers { permanent_uuid: "f649366115124f25bc74e77e773613ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } }
I20260925 16:33:28.096197 8415 heartbeater.cc:503] Master 127.0.98.62:37535 was elected leader, sending a full tablet report...
I20260925 16:33:28.096484 8443 ts_tablet_manager.cc:1428] T bec074d69bcd424392257622110dc61e P 9203d42b2ff045c386ade97cea8d1430: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 16:33:28.152450 8450 raft_consensus.cc:482] T bec074d69bcd424392257622110dc61e P 948c869ed1fe452e8f64517c105e455d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:33:28.152669 8450 raft_consensus.cc:504] T bec074d69bcd424392257622110dc61e P 948c869ed1fe452e8f64517c105e455d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948c869ed1fe452e8f64517c105e455d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41731 } } peers { permanent_uuid: "9203d42b2ff045c386ade97cea8d1430" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36421 } } peers { permanent_uuid: "f649366115124f25bc74e77e773613ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } }
I20260925 16:33:28.153815 8450 leader_election.cc:288] T bec074d69bcd424392257622110dc61e P 948c869ed1fe452e8f64517c105e455d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9203d42b2ff045c386ade97cea8d1430 (127.0.98.3:36421), f649366115124f25bc74e77e773613ad (127.0.98.2:40387)
I20260925 16:33:28.159332 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:28.154153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52765 (local address 127.0.98.3:36421)
0925 16:33:28.154278 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:28.154282 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:28.154607 (+ 325us) server_negotiation.cc:407] Connection header received
0925 16:33:28.154775 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:28.154778 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:28.154830 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:28.154912 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:33:28.155451 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:28.156205 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:28.158041 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:28.158655 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:28.158776 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:28.158842 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:28.159142 (+ 300us) server_negotiation.cc:299] Negotiation successful
0925 16:33:28.159210 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 16:33:28.159363 8453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:28.154035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36421 (local address 127.0.98.1:52765)
0925 16:33:28.154479 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:28.154494 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:28.154651 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:28.154941 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:28.154944 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:28.154982 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:33:28.155233 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:28.155239 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:28.156319 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:28.156323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:28.157916 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:28.157928 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:28.158792 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:28.158796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:28.159007 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:28.159028 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 16:33:28.159127 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 16:33:28.159203 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"mutex_wait_us":221,"spinlock_wait_cycles":27008,"thread_start_us":142,"threads_started":1}
I20260925 16:33:28.159421 8442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:28.154186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60435 (local address 127.0.98.2:40387)
0925 16:33:28.154312 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:28.154318 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:28.154655 (+ 337us) server_negotiation.cc:407] Connection header received
0925 16:33:28.154775 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:28.154778 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:28.154830 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:28.154912 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:33:28.155357 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:28.156205 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:28.158072 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:28.158732 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:28.158842 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:28.158905 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:28.159228 (+ 323us) server_negotiation.cc:299] Negotiation successful
0925 16:33:28.159294 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 16:33:28.159694 8452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:28.154079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40387 (local address 127.0.98.1:60435)
0925 16:33:28.154565 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:28.154601 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:28.154674 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:28.154941 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:28.154944 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:28.154985 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:33:28.155233 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:28.155239 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:28.156319 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:28.156323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:28.157933 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:28.157939 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:28.158965 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:28.158968 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:28.159077 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:28.159100 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 16:33:28.159200 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 16:33:28.159273 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"mutex_wait_us":305,"spinlock_wait_cycles":8448,"thread_start_us":92,"threads_started":1}
I20260925 16:33:28.160060 8369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec074d69bcd424392257622110dc61e" candidate_uuid: "948c869ed1fe452e8f64517c105e455d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9203d42b2ff045c386ade97cea8d1430" is_pre_election: true
I20260925 16:33:28.160301 8369 raft_consensus.cc:2454] T bec074d69bcd424392257622110dc61e P 9203d42b2ff045c386ade97cea8d1430 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948c869ed1fe452e8f64517c105e455d in term 0.
I20260925 16:33:28.160252 8229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec074d69bcd424392257622110dc61e" candidate_uuid: "948c869ed1fe452e8f64517c105e455d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f649366115124f25bc74e77e773613ad" is_pre_election: true
I20260925 16:33:28.160571 8229 raft_consensus.cc:2454] T bec074d69bcd424392257622110dc61e P f649366115124f25bc74e77e773613ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948c869ed1fe452e8f64517c105e455d in term 0.
I20260925 16:33:28.160756 8022 leader_election.cc:302] T bec074d69bcd424392257622110dc61e P 948c869ed1fe452e8f64517c105e455d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9203d42b2ff045c386ade97cea8d1430, 948c869ed1fe452e8f64517c105e455d; no voters:
I20260925 16:33:28.161069 8450 raft_consensus.cc:2781] T bec074d69bcd424392257622110dc61e P 948c869ed1fe452e8f64517c105e455d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:33:28.161164 8450 raft_consensus.cc:482] T bec074d69bcd424392257622110dc61e P 948c869ed1fe452e8f64517c105e455d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:33:28.161226 8450 raft_consensus.cc:3037] T bec074d69bcd424392257622110dc61e P 948c869ed1fe452e8f64517c105e455d [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:33:28.162249 8450 raft_consensus.cc:504] T bec074d69bcd424392257622110dc61e P 948c869ed1fe452e8f64517c105e455d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948c869ed1fe452e8f64517c105e455d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41731 } } peers { permanent_uuid: "9203d42b2ff045c386ade97cea8d1430" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36421 } } peers { permanent_uuid: "f649366115124f25bc74e77e773613ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } }
I20260925 16:33:28.162616 8450 leader_election.cc:288] T bec074d69bcd424392257622110dc61e P 948c869ed1fe452e8f64517c105e455d [CANDIDATE]: Term 1 election: Requested vote from peers 9203d42b2ff045c386ade97cea8d1430 (127.0.98.3:36421), f649366115124f25bc74e77e773613ad (127.0.98.2:40387)
I20260925 16:33:28.162972 8369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec074d69bcd424392257622110dc61e" candidate_uuid: "948c869ed1fe452e8f64517c105e455d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9203d42b2ff045c386ade97cea8d1430"
I20260925 16:33:28.162961 8229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec074d69bcd424392257622110dc61e" candidate_uuid: "948c869ed1fe452e8f64517c105e455d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f649366115124f25bc74e77e773613ad"
I20260925 16:33:28.163082 8229 raft_consensus.cc:3037] T bec074d69bcd424392257622110dc61e P f649366115124f25bc74e77e773613ad [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:33:28.163082 8369 raft_consensus.cc:3037] T bec074d69bcd424392257622110dc61e P 9203d42b2ff045c386ade97cea8d1430 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:33:28.164309 8369 raft_consensus.cc:2454] T bec074d69bcd424392257622110dc61e P 9203d42b2ff045c386ade97cea8d1430 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948c869ed1fe452e8f64517c105e455d in term 1.
I20260925 16:33:28.164309 8229 raft_consensus.cc:2454] T bec074d69bcd424392257622110dc61e P f649366115124f25bc74e77e773613ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948c869ed1fe452e8f64517c105e455d in term 1.
I20260925 16:33:28.164602 8022 leader_election.cc:302] T bec074d69bcd424392257622110dc61e P 948c869ed1fe452e8f64517c105e455d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9203d42b2ff045c386ade97cea8d1430, 948c869ed1fe452e8f64517c105e455d; no voters:
I20260925 16:33:28.164830 8450 raft_consensus.cc:2781] T bec074d69bcd424392257622110dc61e P 948c869ed1fe452e8f64517c105e455d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:33:28.165076 8450 raft_consensus.cc:686] T bec074d69bcd424392257622110dc61e P 948c869ed1fe452e8f64517c105e455d [term 1 LEADER]: Becoming Leader. State: Replica: 948c869ed1fe452e8f64517c105e455d, State: Running, Role: LEADER
I20260925 16:33:28.165431 8450 consensus_queue.cc:237] T bec074d69bcd424392257622110dc61e P 948c869ed1fe452e8f64517c105e455d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948c869ed1fe452e8f64517c105e455d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41731 } } peers { permanent_uuid: "9203d42b2ff045c386ade97cea8d1430" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36421 } } peers { permanent_uuid: "f649366115124f25bc74e77e773613ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } }
I20260925 16:33:28.168583 7932 catalog_manager.cc:5495] T bec074d69bcd424392257622110dc61e P 948c869ed1fe452e8f64517c105e455d reported cstate change: term changed from 0 to 1, leader changed from <none> to 948c869ed1fe452e8f64517c105e455d (127.0.98.1). New cstate: current_term: 1 leader_uuid: "948c869ed1fe452e8f64517c105e455d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948c869ed1fe452e8f64517c105e455d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9203d42b2ff045c386ade97cea8d1430" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f649366115124f25bc74e77e773613ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } health_report { overall_health: UNKNOWN } } }
I20260925 16:33:28.190661 8438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:28.187428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51896 (local address 127.0.98.1:41731)
0925 16:33:28.187576 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:28.187580 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:28.187592 (+ 12us) server_negotiation.cc:407] Connection header received
0925 16:33:28.187664 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:28.187666 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:28.187722 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:28.187815 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:33:28.188035 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:28.188539 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:28.189336 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:28.189555 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:28.189631 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:28.189859 (+ 228us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:33:28.190281 (+ 422us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:33:28.190381 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:28.190427 (+ 46us) server_negotiation.cc:299] Negotiation successful
0925 16:33:28.190486 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 16:33:28.196583 8369 raft_consensus.cc:1261] T bec074d69bcd424392257622110dc61e P 9203d42b2ff045c386ade97cea8d1430 [term 1 FOLLOWER]: Refusing update from remote peer 948c869ed1fe452e8f64517c105e455d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:33:28.196795 8229 raft_consensus.cc:1261] T bec074d69bcd424392257622110dc61e P f649366115124f25bc74e77e773613ad [term 1 FOLLOWER]: Refusing update from remote peer 948c869ed1fe452e8f64517c105e455d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:33:28.197332 8450 consensus_queue.cc:1035] T bec074d69bcd424392257622110dc61e P 948c869ed1fe452e8f64517c105e455d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9203d42b2ff045c386ade97cea8d1430" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:33:28.197535 8454 consensus_queue.cc:1035] T bec074d69bcd424392257622110dc61e P 948c869ed1fe452e8f64517c105e455d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f649366115124f25bc74e77e773613ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:33:28.204670 8457 mvcc.cc:205] Tried to move back new op lower bound from 7333290017565691904 to 7333290017448570880. Current Snapshot: MvccSnapshot[applied={T|T < 7333290017565691904}]
I20260925 16:33:28.205637 8458 mvcc.cc:205] Tried to move back new op lower bound from 7333290017565691904 to 7333290017448570880. Current Snapshot: MvccSnapshot[applied={T|T < 7333290017565691904}]
I20260925 16:33:28.213732 7932 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:33:28.213922 7932 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:33:28.216300 7932 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7bf7b1c2fbb04b7a96e4038edbe73e92: Sending DeleteTablet for 3 replicas of tablet bec074d69bcd424392257622110dc61e
I20260925 16:33:28.217039 8069 tablet_service.cc:1514] Processing DeleteTablet for tablet bec074d69bcd424392257622110dc61e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:33:28 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51892
I20260925 16:33:28.217039 8349 tablet_service.cc:1514] Processing DeleteTablet for tablet bec074d69bcd424392257622110dc61e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:33:28 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58224
I20260925 16:33:28.217154 8209 tablet_service.cc:1514] Processing DeleteTablet for tablet bec074d69bcd424392257622110dc61e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:33:28 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58520
I20260925 16:33:28.217636 8467 tablet_replica.cc:330] T bec074d69bcd424392257622110dc61e P 9203d42b2ff045c386ade97cea8d1430: stopping tablet replica
I20260925 16:33:28.217679 8466 tablet_replica.cc:330] T bec074d69bcd424392257622110dc61e P 948c869ed1fe452e8f64517c105e455d: stopping tablet replica
I20260925 16:33:28.217998 8467 raft_consensus.cc:2229] T bec074d69bcd424392257622110dc61e P 9203d42b2ff045c386ade97cea8d1430 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:33:28.217934 7932 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36614:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 16:33:28.218286 8467 raft_consensus.cc:2258] T bec074d69bcd424392257622110dc61e P 9203d42b2ff045c386ade97cea8d1430 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 16:33:28.218310 7932 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:33:28.218734 8466 raft_consensus.cc:2229] T bec074d69bcd424392257622110dc61e P 948c869ed1fe452e8f64517c105e455d [term 1 LEADER]: Raft consensus shutting down.
I20260925 16:33:28.219166 8466 raft_consensus.cc:2258] T bec074d69bcd424392257622110dc61e P 948c869ed1fe452e8f64517c105e455d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:33:28.219588 8468 tablet_replica.cc:330] T bec074d69bcd424392257622110dc61e P f649366115124f25bc74e77e773613ad: stopping tablet replica
I20260925 16:33:28.220858 8467 ts_tablet_manager.cc:1905] T bec074d69bcd424392257622110dc61e P 9203d42b2ff045c386ade97cea8d1430: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:33:28.221413 8466 ts_tablet_manager.cc:1905] T bec074d69bcd424392257622110dc61e P 948c869ed1fe452e8f64517c105e455d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:33:28.222895 8069 tablet_service.cc:1467] Processing CreateTablet for tablet 8fe0c09b6acc48b2b397fc423637aea3 (DEFAULT_TABLE table=test-table [id=be7a7c20356c4aa0b2c3aa9fe3c3c3f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:33:28.223007 8349 tablet_service.cc:1467] Processing CreateTablet for tablet 8fe0c09b6acc48b2b397fc423637aea3 (DEFAULT_TABLE table=test-table [id=be7a7c20356c4aa0b2c3aa9fe3c3c3f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:33:28.223169 8069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fe0c09b6acc48b2b397fc423637aea3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:33:28.223275 8349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fe0c09b6acc48b2b397fc423637aea3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:33:28.225040 8445 tablet_bootstrap.cc:492] T 8fe0c09b6acc48b2b397fc423637aea3 P 948c869ed1fe452e8f64517c105e455d: Bootstrap starting.
I20260925 16:33:28.225149 8466 ts_tablet_manager.cc:1918] T bec074d69bcd424392257622110dc61e P 948c869ed1fe452e8f64517c105e455d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:33:28.225238 8466 log.cc:1196] T bec074d69bcd424392257622110dc61e P 948c869ed1fe452e8f64517c105e455d: Deleting WAL directory at /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/ts-0/wal/wals/bec074d69bcd424392257622110dc61e
I20260925 16:33:28.225569 8466 ts_tablet_manager.cc:1939] T bec074d69bcd424392257622110dc61e P 948c869ed1fe452e8f64517c105e455d: Deleting consensus metadata
I20260925 16:33:28.226007 8445 tablet_bootstrap.cc:654] T 8fe0c09b6acc48b2b397fc423637aea3 P 948c869ed1fe452e8f64517c105e455d: Neither blocks nor log segments found. Creating new log.
I20260925 16:33:28.226533 8467 ts_tablet_manager.cc:1918] T bec074d69bcd424392257622110dc61e P 9203d42b2ff045c386ade97cea8d1430: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:33:28.226634 8443 tablet_bootstrap.cc:492] T 8fe0c09b6acc48b2b397fc423637aea3 P 9203d42b2ff045c386ade97cea8d1430: Bootstrap starting.
I20260925 16:33:28.219880 8468 raft_consensus.cc:2229] T bec074d69bcd424392257622110dc61e P f649366115124f25bc74e77e773613ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:33:28.223059 8209 tablet_service.cc:1467] Processing CreateTablet for tablet 8fe0c09b6acc48b2b397fc423637aea3 (DEFAULT_TABLE table=test-table [id=be7a7c20356c4aa0b2c3aa9fe3c3c3f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:33:28.226981 8209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fe0c09b6acc48b2b397fc423637aea3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:33:28.227022 8468 raft_consensus.cc:2258] T bec074d69bcd424392257622110dc61e P f649366115124f25bc74e77e773613ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:33:28.226639 8467 log.cc:1196] T bec074d69bcd424392257622110dc61e P 9203d42b2ff045c386ade97cea8d1430: Deleting WAL directory at /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/ts-2/wal/wals/bec074d69bcd424392257622110dc61e
I20260925 16:33:28.227555 8467 ts_tablet_manager.cc:1939] T bec074d69bcd424392257622110dc61e P 9203d42b2ff045c386ade97cea8d1430: Deleting consensus metadata
I20260925 16:33:28.227628 8443 tablet_bootstrap.cc:654] T 8fe0c09b6acc48b2b397fc423637aea3 P 9203d42b2ff045c386ade97cea8d1430: Neither blocks nor log segments found. Creating new log.
I20260925 16:33:28.228703 7918 catalog_manager.cc:4841] TS 9203d42b2ff045c386ade97cea8d1430 (127.0.98.3:36421): tablet bec074d69bcd424392257622110dc61e (table test-table [id=dfbe188c5ec0421ba7b42537fbae9bd5]) successfully deleted
I20260925 16:33:28.229312 7917 catalog_manager.cc:4841] TS 948c869ed1fe452e8f64517c105e455d (127.0.98.1:41731): tablet bec074d69bcd424392257622110dc61e (table test-table [id=dfbe188c5ec0421ba7b42537fbae9bd5]) successfully deleted
I20260925 16:33:28.229539 8444 tablet_bootstrap.cc:492] T 8fe0c09b6acc48b2b397fc423637aea3 P f649366115124f25bc74e77e773613ad: Bootstrap starting.
I20260925 16:33:28.230345 8444 tablet_bootstrap.cc:654] T 8fe0c09b6acc48b2b397fc423637aea3 P f649366115124f25bc74e77e773613ad: Neither blocks nor log segments found. Creating new log.
I20260925 16:33:28.231634 8445 tablet_bootstrap.cc:492] T 8fe0c09b6acc48b2b397fc423637aea3 P 948c869ed1fe452e8f64517c105e455d: No bootstrap required, opened a new log
I20260925 16:33:28.231665 8443 tablet_bootstrap.cc:492] T 8fe0c09b6acc48b2b397fc423637aea3 P 9203d42b2ff045c386ade97cea8d1430: No bootstrap required, opened a new log
I20260925 16:33:28.231736 8443 ts_tablet_manager.cc:1397] T 8fe0c09b6acc48b2b397fc423637aea3 P 9203d42b2ff045c386ade97cea8d1430: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260925 16:33:28.231739 8445 ts_tablet_manager.cc:1397] T 8fe0c09b6acc48b2b397fc423637aea3 P 948c869ed1fe452e8f64517c105e455d: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20260925 16:33:28.232151 8443 raft_consensus.cc:348] T 8fe0c09b6acc48b2b397fc423637aea3 P 9203d42b2ff045c386ade97cea8d1430 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948c869ed1fe452e8f64517c105e455d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41731 } } peers { permanent_uuid: "f649366115124f25bc74e77e773613ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } } peers { permanent_uuid: "9203d42b2ff045c386ade97cea8d1430" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36421 } }
I20260925 16:33:28.232169 8445 raft_consensus.cc:348] T 8fe0c09b6acc48b2b397fc423637aea3 P 948c869ed1fe452e8f64517c105e455d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948c869ed1fe452e8f64517c105e455d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41731 } } peers { permanent_uuid: "f649366115124f25bc74e77e773613ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } } peers { permanent_uuid: "9203d42b2ff045c386ade97cea8d1430" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36421 } }
I20260925 16:33:28.232251 8443 raft_consensus.cc:374] T 8fe0c09b6acc48b2b397fc423637aea3 P 9203d42b2ff045c386ade97cea8d1430 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:33:28.232272 8445 raft_consensus.cc:374] T 8fe0c09b6acc48b2b397fc423637aea3 P 948c869ed1fe452e8f64517c105e455d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:33:28.232282 8443 raft_consensus.cc:729] T 8fe0c09b6acc48b2b397fc423637aea3 P 9203d42b2ff045c386ade97cea8d1430 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9203d42b2ff045c386ade97cea8d1430, State: Initialized, Role: FOLLOWER
I20260925 16:33:28.232302 8445 raft_consensus.cc:729] T 8fe0c09b6acc48b2b397fc423637aea3 P 948c869ed1fe452e8f64517c105e455d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948c869ed1fe452e8f64517c105e455d, State: Initialized, Role: FOLLOWER
I20260925 16:33:28.232354 8468 ts_tablet_manager.cc:1905] T bec074d69bcd424392257622110dc61e P f649366115124f25bc74e77e773613ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:33:28.232400 8443 consensus_queue.cc:260] T 8fe0c09b6acc48b2b397fc423637aea3 P 9203d42b2ff045c386ade97cea8d1430 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948c869ed1fe452e8f64517c105e455d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41731 } } peers { permanent_uuid: "f649366115124f25bc74e77e773613ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } } peers { permanent_uuid: "9203d42b2ff045c386ade97cea8d1430" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36421 } }
I20260925 16:33:28.232429 8445 consensus_queue.cc:260] T 8fe0c09b6acc48b2b397fc423637aea3 P 948c869ed1fe452e8f64517c105e455d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948c869ed1fe452e8f64517c105e455d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41731 } } peers { permanent_uuid: "f649366115124f25bc74e77e773613ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } } peers { permanent_uuid: "9203d42b2ff045c386ade97cea8d1430" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36421 } }
I20260925 16:33:28.232630 8443 ts_tablet_manager.cc:1428] T 8fe0c09b6acc48b2b397fc423637aea3 P 9203d42b2ff045c386ade97cea8d1430: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 16:33:28.232678 8445 ts_tablet_manager.cc:1428] T 8fe0c09b6acc48b2b397fc423637aea3 P 948c869ed1fe452e8f64517c105e455d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 16:33:28.234243 8444 tablet_bootstrap.cc:492] T 8fe0c09b6acc48b2b397fc423637aea3 P f649366115124f25bc74e77e773613ad: No bootstrap required, opened a new log
I20260925 16:33:28.234324 8444 ts_tablet_manager.cc:1397] T 8fe0c09b6acc48b2b397fc423637aea3 P f649366115124f25bc74e77e773613ad: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260925 16:33:28.234805 8444 raft_consensus.cc:348] T 8fe0c09b6acc48b2b397fc423637aea3 P f649366115124f25bc74e77e773613ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948c869ed1fe452e8f64517c105e455d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41731 } } peers { permanent_uuid: "f649366115124f25bc74e77e773613ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } } peers { permanent_uuid: "9203d42b2ff045c386ade97cea8d1430" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36421 } }
I20260925 16:33:28.234930 8444 raft_consensus.cc:374] T 8fe0c09b6acc48b2b397fc423637aea3 P f649366115124f25bc74e77e773613ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:33:28.234972 8444 raft_consensus.cc:729] T 8fe0c09b6acc48b2b397fc423637aea3 P f649366115124f25bc74e77e773613ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f649366115124f25bc74e77e773613ad, State: Initialized, Role: FOLLOWER
I20260925 16:33:28.235096 8444 consensus_queue.cc:260] T 8fe0c09b6acc48b2b397fc423637aea3 P f649366115124f25bc74e77e773613ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948c869ed1fe452e8f64517c105e455d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41731 } } peers { permanent_uuid: "f649366115124f25bc74e77e773613ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } } peers { permanent_uuid: "9203d42b2ff045c386ade97cea8d1430" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36421 } }
I20260925 16:33:28.235350 8444 ts_tablet_manager.cc:1428] T 8fe0c09b6acc48b2b397fc423637aea3 P f649366115124f25bc74e77e773613ad: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 16:33:28.236683 8468 ts_tablet_manager.cc:1918] T bec074d69bcd424392257622110dc61e P f649366115124f25bc74e77e773613ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:33:28.236796 8468 log.cc:1196] T bec074d69bcd424392257622110dc61e P f649366115124f25bc74e77e773613ad: Deleting WAL directory at /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/ts-1/wal/wals/bec074d69bcd424392257622110dc61e
I20260925 16:33:28.237124 8468 ts_tablet_manager.cc:1939] T bec074d69bcd424392257622110dc61e P f649366115124f25bc74e77e773613ad: Deleting consensus metadata
I20260925 16:33:28.238190 7916 catalog_manager.cc:4841] TS f649366115124f25bc74e77e773613ad (127.0.98.2:40387): tablet bec074d69bcd424392257622110dc61e (table test-table [id=dfbe188c5ec0421ba7b42537fbae9bd5]) successfully deleted
W20260925 16:33:28.259336 8416 tablet.cc:2369] T 8fe0c09b6acc48b2b397fc423637aea3 P 9203d42b2ff045c386ade97cea8d1430: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 16:33:28.282945 8276 tablet.cc:2369] T 8fe0c09b6acc48b2b397fc423637aea3 P f649366115124f25bc74e77e773613ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 16:33:28.323143 8136 tablet.cc:2369] T 8fe0c09b6acc48b2b397fc423637aea3 P 948c869ed1fe452e8f64517c105e455d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:33:28.491374 8454 raft_consensus.cc:482] T 8fe0c09b6acc48b2b397fc423637aea3 P 948c869ed1fe452e8f64517c105e455d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:33:28.491536 8454 raft_consensus.cc:504] T 8fe0c09b6acc48b2b397fc423637aea3 P 948c869ed1fe452e8f64517c105e455d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948c869ed1fe452e8f64517c105e455d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41731 } } peers { permanent_uuid: "f649366115124f25bc74e77e773613ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } } peers { permanent_uuid: "9203d42b2ff045c386ade97cea8d1430" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36421 } }
I20260925 16:33:28.492043 8454 leader_election.cc:288] T 8fe0c09b6acc48b2b397fc423637aea3 P 948c869ed1fe452e8f64517c105e455d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f649366115124f25bc74e77e773613ad (127.0.98.2:40387), 9203d42b2ff045c386ade97cea8d1430 (127.0.98.3:36421)
I20260925 16:33:28.492493 8229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fe0c09b6acc48b2b397fc423637aea3" candidate_uuid: "948c869ed1fe452e8f64517c105e455d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f649366115124f25bc74e77e773613ad" is_pre_election: true
I20260925 16:33:28.492623 8229 raft_consensus.cc:2454] T 8fe0c09b6acc48b2b397fc423637aea3 P f649366115124f25bc74e77e773613ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948c869ed1fe452e8f64517c105e455d in term 0.
I20260925 16:33:28.492637 8368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fe0c09b6acc48b2b397fc423637aea3" candidate_uuid: "948c869ed1fe452e8f64517c105e455d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9203d42b2ff045c386ade97cea8d1430" is_pre_election: true
I20260925 16:33:28.492868 8368 raft_consensus.cc:2454] T 8fe0c09b6acc48b2b397fc423637aea3 P 9203d42b2ff045c386ade97cea8d1430 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948c869ed1fe452e8f64517c105e455d in term 0.
I20260925 16:33:28.492961 8020 leader_election.cc:302] T 8fe0c09b6acc48b2b397fc423637aea3 P 948c869ed1fe452e8f64517c105e455d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 948c869ed1fe452e8f64517c105e455d, f649366115124f25bc74e77e773613ad; no voters:
I20260925 16:33:28.493136 8454 raft_consensus.cc:2781] T 8fe0c09b6acc48b2b397fc423637aea3 P 948c869ed1fe452e8f64517c105e455d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:33:28.493194 8454 raft_consensus.cc:482] T 8fe0c09b6acc48b2b397fc423637aea3 P 948c869ed1fe452e8f64517c105e455d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:33:28.493242 8454 raft_consensus.cc:3037] T 8fe0c09b6acc48b2b397fc423637aea3 P 948c869ed1fe452e8f64517c105e455d [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:33:28.494241 8454 raft_consensus.cc:504] T 8fe0c09b6acc48b2b397fc423637aea3 P 948c869ed1fe452e8f64517c105e455d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948c869ed1fe452e8f64517c105e455d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41731 } } peers { permanent_uuid: "f649366115124f25bc74e77e773613ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } } peers { permanent_uuid: "9203d42b2ff045c386ade97cea8d1430" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36421 } }
I20260925 16:33:28.494656 8454 leader_election.cc:288] T 8fe0c09b6acc48b2b397fc423637aea3 P 948c869ed1fe452e8f64517c105e455d [CANDIDATE]: Term 1 election: Requested vote from peers f649366115124f25bc74e77e773613ad (127.0.98.2:40387), 9203d42b2ff045c386ade97cea8d1430 (127.0.98.3:36421)
I20260925 16:33:28.494943 8229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fe0c09b6acc48b2b397fc423637aea3" candidate_uuid: "948c869ed1fe452e8f64517c105e455d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f649366115124f25bc74e77e773613ad"
I20260925 16:33:28.495078 8229 raft_consensus.cc:3037] T 8fe0c09b6acc48b2b397fc423637aea3 P f649366115124f25bc74e77e773613ad [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:33:28.495086 8368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fe0c09b6acc48b2b397fc423637aea3" candidate_uuid: "948c869ed1fe452e8f64517c105e455d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9203d42b2ff045c386ade97cea8d1430"
I20260925 16:33:28.495235 8368 raft_consensus.cc:3037] T 8fe0c09b6acc48b2b397fc423637aea3 P 9203d42b2ff045c386ade97cea8d1430 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:33:28.496133 8229 raft_consensus.cc:2454] T 8fe0c09b6acc48b2b397fc423637aea3 P f649366115124f25bc74e77e773613ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948c869ed1fe452e8f64517c105e455d in term 1.
I20260925 16:33:28.496397 8020 leader_election.cc:302] T 8fe0c09b6acc48b2b397fc423637aea3 P 948c869ed1fe452e8f64517c105e455d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 948c869ed1fe452e8f64517c105e455d, f649366115124f25bc74e77e773613ad; no voters:
I20260925 16:33:28.496475 8368 raft_consensus.cc:2454] T 8fe0c09b6acc48b2b397fc423637aea3 P 9203d42b2ff045c386ade97cea8d1430 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948c869ed1fe452e8f64517c105e455d in term 1.
I20260925 16:33:28.496567 8454 raft_consensus.cc:2781] T 8fe0c09b6acc48b2b397fc423637aea3 P 948c869ed1fe452e8f64517c105e455d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:33:28.496662 8454 raft_consensus.cc:686] T 8fe0c09b6acc48b2b397fc423637aea3 P 948c869ed1fe452e8f64517c105e455d [term 1 LEADER]: Becoming Leader. State: Replica: 948c869ed1fe452e8f64517c105e455d, State: Running, Role: LEADER
I20260925 16:33:28.496816 8454 consensus_queue.cc:237] T 8fe0c09b6acc48b2b397fc423637aea3 P 948c869ed1fe452e8f64517c105e455d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948c869ed1fe452e8f64517c105e455d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41731 } } peers { permanent_uuid: "f649366115124f25bc74e77e773613ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } } peers { permanent_uuid: "9203d42b2ff045c386ade97cea8d1430" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36421 } }
I20260925 16:33:28.498395 7931 catalog_manager.cc:5495] T 8fe0c09b6acc48b2b397fc423637aea3 P 948c869ed1fe452e8f64517c105e455d reported cstate change: term changed from 0 to 1, leader changed from <none> to 948c869ed1fe452e8f64517c105e455d (127.0.98.1). New cstate: current_term: 1 leader_uuid: "948c869ed1fe452e8f64517c105e455d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948c869ed1fe452e8f64517c105e455d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f649366115124f25bc74e77e773613ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9203d42b2ff045c386ade97cea8d1430" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36421 } health_report { overall_health: UNKNOWN } } }
Sep 25 16:33:28 dist-test-slave-65pf krb5kdc[7887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354007, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 16:33:28.556867 8478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:28.548357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37535 (local address 127.0.0.1:36616)
0925 16:33:28.548674 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:28.548687 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:28.548831 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:28.549089 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:28.549092 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:28.549524 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:28.549615 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:28.549622 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:28.550352 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:28.550358 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:28.550969 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:28.550977 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:28.551419 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:28.551423 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:28.551497 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:28.552082 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:28.552094 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:28.553535 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:33:28.555318 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:28.555325 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:28.555331 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:28.555605 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:28.555891 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:28.555893 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:28.555894 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:28.556007 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:28.556381 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:33:28.556388 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:33:28.556491 (+ 103us) client_negotiation.cc:769] Sending connection context
0925 16:33:28.556582 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 16:33:28.556689 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":110,"threads_started":1}
I20260925 16:33:28.556874 7981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:28.548367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36616 (local address 127.0.98.62:37535)
0925 16:33:28.548488 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:28.548492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:28.548767 (+ 275us) server_negotiation.cc:407] Connection header received
0925 16:33:28.548929 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:28.548933 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:28.548979 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:28.549069 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:28.549730 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:28.550246 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:28.551092 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:28.551312 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:28.551416 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:28.553687 (+ 2271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:28.553690 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:28.553692 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:28.553715 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:28.555197 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:28.555725 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:28.555728 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:28.555729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:28.555803 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:28.556086 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:28.556088 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:28.556089 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:28.556291 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:28.556368 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:28.556650 (+ 282us) server_negotiation.cc:299] Negotiation successful
0925 16:33:28.556749 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 16:33:28.560855 7931 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36616:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 16:33:28.561493 7931 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:33:28.566615 8069 tablet_service.cc:1467] Processing CreateTablet for tablet a636caf550ff41389999542e067c9a4a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c566584313dc44d7a6ddc62ecefbb1b4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 16:33:28.566695 8209 tablet_service.cc:1467] Processing CreateTablet for tablet a636caf550ff41389999542e067c9a4a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c566584313dc44d7a6ddc62ecefbb1b4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 16:33:28.567066 8069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a636caf550ff41389999542e067c9a4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:33:28.567076 8209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a636caf550ff41389999542e067c9a4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:33:28.566991 8349 tablet_service.cc:1467] Processing CreateTablet for tablet a636caf550ff41389999542e067c9a4a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c566584313dc44d7a6ddc62ecefbb1b4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 16:33:28.567328 8349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a636caf550ff41389999542e067c9a4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:33:28.569670 8444 tablet_bootstrap.cc:492] T a636caf550ff41389999542e067c9a4a P f649366115124f25bc74e77e773613ad: Bootstrap starting.
I20260925 16:33:28.570003 8445 tablet_bootstrap.cc:492] T a636caf550ff41389999542e067c9a4a P 948c869ed1fe452e8f64517c105e455d: Bootstrap starting.
I20260925 16:33:28.570235 8443 tablet_bootstrap.cc:492] T a636caf550ff41389999542e067c9a4a P 9203d42b2ff045c386ade97cea8d1430: Bootstrap starting.
I20260925 16:33:28.570703 8444 tablet_bootstrap.cc:654] T a636caf550ff41389999542e067c9a4a P f649366115124f25bc74e77e773613ad: Neither blocks nor log segments found. Creating new log.
I20260925 16:33:28.570904 8445 tablet_bootstrap.cc:654] T a636caf550ff41389999542e067c9a4a P 948c869ed1fe452e8f64517c105e455d: Neither blocks nor log segments found. Creating new log.
I20260925 16:33:28.571331 8443 tablet_bootstrap.cc:654] T a636caf550ff41389999542e067c9a4a P 9203d42b2ff045c386ade97cea8d1430: Neither blocks nor log segments found. Creating new log.
I20260925 16:33:28.572345 8444 tablet_bootstrap.cc:492] T a636caf550ff41389999542e067c9a4a P f649366115124f25bc74e77e773613ad: No bootstrap required, opened a new log
I20260925 16:33:28.572448 8444 ts_tablet_manager.cc:1397] T a636caf550ff41389999542e067c9a4a P f649366115124f25bc74e77e773613ad: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260925 16:33:28.572769 8445 tablet_bootstrap.cc:492] T a636caf550ff41389999542e067c9a4a P 948c869ed1fe452e8f64517c105e455d: No bootstrap required, opened a new log
I20260925 16:33:28.572917 8445 ts_tablet_manager.cc:1397] T a636caf550ff41389999542e067c9a4a P 948c869ed1fe452e8f64517c105e455d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 16:33:28.572873 8444 raft_consensus.cc:348] T a636caf550ff41389999542e067c9a4a P f649366115124f25bc74e77e773613ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948c869ed1fe452e8f64517c105e455d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41731 } } peers { permanent_uuid: "f649366115124f25bc74e77e773613ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } } peers { permanent_uuid: "9203d42b2ff045c386ade97cea8d1430" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36421 } }
I20260925 16:33:28.572983 8444 raft_consensus.cc:374] T a636caf550ff41389999542e067c9a4a P f649366115124f25bc74e77e773613ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:33:28.573019 8444 raft_consensus.cc:729] T a636caf550ff41389999542e067c9a4a P f649366115124f25bc74e77e773613ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f649366115124f25bc74e77e773613ad, State: Initialized, Role: FOLLOWER
I20260925 16:33:28.573136 8444 consensus_queue.cc:260] T a636caf550ff41389999542e067c9a4a P f649366115124f25bc74e77e773613ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948c869ed1fe452e8f64517c105e455d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41731 } } peers { permanent_uuid: "f649366115124f25bc74e77e773613ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } } peers { permanent_uuid: "9203d42b2ff045c386ade97cea8d1430" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36421 } }
I20260925 16:33:28.573374 8444 ts_tablet_manager.cc:1428] T a636caf550ff41389999542e067c9a4a P f649366115124f25bc74e77e773613ad: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260925 16:33:28.573405 8445 raft_consensus.cc:348] T a636caf550ff41389999542e067c9a4a P 948c869ed1fe452e8f64517c105e455d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948c869ed1fe452e8f64517c105e455d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41731 } } peers { permanent_uuid: "f649366115124f25bc74e77e773613ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } } peers { permanent_uuid: "9203d42b2ff045c386ade97cea8d1430" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36421 } }
I20260925 16:33:28.573514 8445 raft_consensus.cc:374] T a636caf550ff41389999542e067c9a4a P 948c869ed1fe452e8f64517c105e455d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:33:28.573527 8443 tablet_bootstrap.cc:492] T a636caf550ff41389999542e067c9a4a P 9203d42b2ff045c386ade97cea8d1430: No bootstrap required, opened a new log
I20260925 16:33:28.573553 8445 raft_consensus.cc:729] T a636caf550ff41389999542e067c9a4a P 948c869ed1fe452e8f64517c105e455d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948c869ed1fe452e8f64517c105e455d, State: Initialized, Role: FOLLOWER
I20260925 16:33:28.573606 8443 ts_tablet_manager.cc:1397] T a636caf550ff41389999542e067c9a4a P 9203d42b2ff045c386ade97cea8d1430: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 16:33:28.573685 8445 consensus_queue.cc:260] T a636caf550ff41389999542e067c9a4a P 948c869ed1fe452e8f64517c105e455d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948c869ed1fe452e8f64517c105e455d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41731 } } peers { permanent_uuid: "f649366115124f25bc74e77e773613ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } } peers { permanent_uuid: "9203d42b2ff045c386ade97cea8d1430" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36421 } }
I20260925 16:33:28.573961 8445 ts_tablet_manager.cc:1428] T a636caf550ff41389999542e067c9a4a P 948c869ed1fe452e8f64517c105e455d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 16:33:28.574056 8443 raft_consensus.cc:348] T a636caf550ff41389999542e067c9a4a P 9203d42b2ff045c386ade97cea8d1430 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948c869ed1fe452e8f64517c105e455d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41731 } } peers { permanent_uuid: "f649366115124f25bc74e77e773613ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } } peers { permanent_uuid: "9203d42b2ff045c386ade97cea8d1430" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36421 } }
I20260925 16:33:28.574116 8454 tablet_replica.cc:439] T a636caf550ff41389999542e067c9a4a P 948c869ed1fe452e8f64517c105e455d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948c869ed1fe452e8f64517c105e455d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41731 } } peers { permanent_uuid: "f649366115124f25bc74e77e773613ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } } peers { permanent_uuid: "9203d42b2ff045c386ade97cea8d1430" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36421 } } }
I20260925 16:33:28.574174 8443 raft_consensus.cc:374] T a636caf550ff41389999542e067c9a4a P 9203d42b2ff045c386ade97cea8d1430 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:33:28.574230 8454 tablet_replica.cc:442] T a636caf550ff41389999542e067c9a4a P 948c869ed1fe452e8f64517c105e455d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 16:33:28.574229 8443 raft_consensus.cc:729] T a636caf550ff41389999542e067c9a4a P 9203d42b2ff045c386ade97cea8d1430 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9203d42b2ff045c386ade97cea8d1430, State: Initialized, Role: FOLLOWER
I20260925 16:33:28.574332 8443 consensus_queue.cc:260] T a636caf550ff41389999542e067c9a4a P 9203d42b2ff045c386ade97cea8d1430 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948c869ed1fe452e8f64517c105e455d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41731 } } peers { permanent_uuid: "f649366115124f25bc74e77e773613ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } } peers { permanent_uuid: "9203d42b2ff045c386ade97cea8d1430" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36421 } }
I20260925 16:33:28.574352 8449 tablet_replica.cc:439] T a636caf550ff41389999542e067c9a4a P f649366115124f25bc74e77e773613ad: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948c869ed1fe452e8f64517c105e455d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41731 } } peers { permanent_uuid: "f649366115124f25bc74e77e773613ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } } peers { permanent_uuid: "9203d42b2ff045c386ade97cea8d1430" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36421 } } }
I20260925 16:33:28.574473 8449 tablet_replica.cc:442] T a636caf550ff41389999542e067c9a4a P f649366115124f25bc74e77e773613ad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 16:33:28.574651 8443 ts_tablet_manager.cc:1428] T a636caf550ff41389999542e067c9a4a P 9203d42b2ff045c386ade97cea8d1430: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 16:33:28.574550 8451 tablet_replica.cc:439] T a636caf550ff41389999542e067c9a4a P 9203d42b2ff045c386ade97cea8d1430: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948c869ed1fe452e8f64517c105e455d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41731 } } peers { permanent_uuid: "f649366115124f25bc74e77e773613ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } } peers { permanent_uuid: "9203d42b2ff045c386ade97cea8d1430" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36421 } } }
I20260925 16:33:28.574726 8451 tablet_replica.cc:442] T a636caf550ff41389999542e067c9a4a P 9203d42b2ff045c386ade97cea8d1430: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 16:33:28.736728 8451 raft_consensus.cc:482] T a636caf550ff41389999542e067c9a4a P 9203d42b2ff045c386ade97cea8d1430 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:33:28.736896 8451 raft_consensus.cc:504] T a636caf550ff41389999542e067c9a4a P 9203d42b2ff045c386ade97cea8d1430 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948c869ed1fe452e8f64517c105e455d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41731 } } peers { permanent_uuid: "f649366115124f25bc74e77e773613ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } } peers { permanent_uuid: "9203d42b2ff045c386ade97cea8d1430" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36421 } }
I20260925 16:33:28.738003 8451 leader_election.cc:288] T a636caf550ff41389999542e067c9a4a P 9203d42b2ff045c386ade97cea8d1430 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948c869ed1fe452e8f64517c105e455d (127.0.98.1:41731), f649366115124f25bc74e77e773613ad (127.0.98.2:40387)
I20260925 16:33:28.741837 8482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:28.738235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41731 (local address 127.0.98.3:58927)
0925 16:33:28.738538 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:28.738555 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:28.738694 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:28.738980 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:28.738983 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:28.739000 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:33:28.739107 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:28.739112 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:28.739832 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:28.739837 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:28.740740 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:28.740750 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:28.741406 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:28.741410 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:28.741508 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:28.741527 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 16:33:28.741615 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 16:33:28.741685 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":79,"threads_started":1}
I20260925 16:33:28.741854 8483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:28.738300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58927 (local address 127.0.98.1:41731)
0925 16:33:28.738541 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:28.738546 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:28.738653 (+ 107us) server_negotiation.cc:407] Connection header received
0925 16:33:28.738821 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:28.738823 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:28.738865 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:28.738970 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:33:28.739215 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:28.739725 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:28.740861 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:28.741281 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:28.741398 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:28.741461 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:28.741630 (+ 169us) server_negotiation.cc:299] Negotiation successful
0925 16:33:28.741699 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":69,"threads_started":1}
I20260925 16:33:28.741914 8484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:28.738342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48969 (local address 127.0.98.2:40387)
0925 16:33:28.738635 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:28.738640 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:28.738838 (+ 198us) server_negotiation.cc:407] Connection header received
0925 16:33:28.738993 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:28.738996 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:28.739046 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:28.739139 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:33:28.739390 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:28.739882 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:28.740942 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:28.741362 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:28.741481 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:28.741544 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:28.741709 (+ 165us) server_negotiation.cc:299] Negotiation successful
0925 16:33:28.741777 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":106,"threads_started":1}
I20260925 16:33:28.741955 8485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:28.738238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40387 (local address 127.0.98.3:48969)
0925 16:33:28.738739 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:28.738751 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:28.738883 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:28.739147 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:28.739150 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:28.739165 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:33:28.739281 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:28.739286 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:28.739987 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:28.739994 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:28.740823 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:28.740832 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:28.741481 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:28.741485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:28.741581 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:28.741608 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 16:33:28.741695 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 16:33:28.741765 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"mutex_wait_us":225,"thread_start_us":41,"threads_started":1}
I20260925 16:33:28.742529 8229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a636caf550ff41389999542e067c9a4a" candidate_uuid: "9203d42b2ff045c386ade97cea8d1430" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f649366115124f25bc74e77e773613ad" is_pre_election: true
I20260925 16:33:28.742620 8089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a636caf550ff41389999542e067c9a4a" candidate_uuid: "9203d42b2ff045c386ade97cea8d1430" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948c869ed1fe452e8f64517c105e455d" is_pre_election: true
I20260925 16:33:28.742738 8229 raft_consensus.cc:2454] T a636caf550ff41389999542e067c9a4a P f649366115124f25bc74e77e773613ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9203d42b2ff045c386ade97cea8d1430 in term 0.
I20260925 16:33:28.742877 8089 raft_consensus.cc:2454] T a636caf550ff41389999542e067c9a4a P 948c869ed1fe452e8f64517c105e455d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9203d42b2ff045c386ade97cea8d1430 in term 0.
I20260925 16:33:28.743088 8300 leader_election.cc:302] T a636caf550ff41389999542e067c9a4a P 9203d42b2ff045c386ade97cea8d1430 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9203d42b2ff045c386ade97cea8d1430, f649366115124f25bc74e77e773613ad; no voters:
I20260925 16:33:28.743360 8451 raft_consensus.cc:2781] T a636caf550ff41389999542e067c9a4a P 9203d42b2ff045c386ade97cea8d1430 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:33:28.743448 8451 raft_consensus.cc:482] T a636caf550ff41389999542e067c9a4a P 9203d42b2ff045c386ade97cea8d1430 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:33:28.743501 8451 raft_consensus.cc:3037] T a636caf550ff41389999542e067c9a4a P 9203d42b2ff045c386ade97cea8d1430 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:33:28.744546 8451 raft_consensus.cc:504] T a636caf550ff41389999542e067c9a4a P 9203d42b2ff045c386ade97cea8d1430 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948c869ed1fe452e8f64517c105e455d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41731 } } peers { permanent_uuid: "f649366115124f25bc74e77e773613ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } } peers { permanent_uuid: "9203d42b2ff045c386ade97cea8d1430" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36421 } }
I20260925 16:33:28.744926 8451 leader_election.cc:288] T a636caf550ff41389999542e067c9a4a P 9203d42b2ff045c386ade97cea8d1430 [CANDIDATE]: Term 1 election: Requested vote from peers 948c869ed1fe452e8f64517c105e455d (127.0.98.1:41731), f649366115124f25bc74e77e773613ad (127.0.98.2:40387)
I20260925 16:33:28.745242 8089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a636caf550ff41389999542e067c9a4a" candidate_uuid: "9203d42b2ff045c386ade97cea8d1430" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948c869ed1fe452e8f64517c105e455d"
I20260925 16:33:28.745262 8229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a636caf550ff41389999542e067c9a4a" candidate_uuid: "9203d42b2ff045c386ade97cea8d1430" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f649366115124f25bc74e77e773613ad"
I20260925 16:33:28.745362 8089 raft_consensus.cc:3037] T a636caf550ff41389999542e067c9a4a P 948c869ed1fe452e8f64517c105e455d [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:33:28.745381 8229 raft_consensus.cc:3037] T a636caf550ff41389999542e067c9a4a P f649366115124f25bc74e77e773613ad [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:33:28.746420 8089 raft_consensus.cc:2454] T a636caf550ff41389999542e067c9a4a P 948c869ed1fe452e8f64517c105e455d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9203d42b2ff045c386ade97cea8d1430 in term 1.
I20260925 16:33:28.746420 8229 raft_consensus.cc:2454] T a636caf550ff41389999542e067c9a4a P f649366115124f25bc74e77e773613ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9203d42b2ff045c386ade97cea8d1430 in term 1.
I20260925 16:33:28.746693 8300 leader_election.cc:302] T a636caf550ff41389999542e067c9a4a P 9203d42b2ff045c386ade97cea8d1430 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9203d42b2ff045c386ade97cea8d1430, f649366115124f25bc74e77e773613ad; no voters:
I20260925 16:33:28.746869 8451 raft_consensus.cc:2781] T a636caf550ff41389999542e067c9a4a P 9203d42b2ff045c386ade97cea8d1430 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:33:28.747035 8451 raft_consensus.cc:686] T a636caf550ff41389999542e067c9a4a P 9203d42b2ff045c386ade97cea8d1430 [term 1 LEADER]: Becoming Leader. State: Replica: 9203d42b2ff045c386ade97cea8d1430, State: Running, Role: LEADER
I20260925 16:33:28.747273 8451 consensus_queue.cc:237] T a636caf550ff41389999542e067c9a4a P 9203d42b2ff045c386ade97cea8d1430 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948c869ed1fe452e8f64517c105e455d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41731 } } peers { permanent_uuid: "f649366115124f25bc74e77e773613ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } } peers { permanent_uuid: "9203d42b2ff045c386ade97cea8d1430" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36421 } }
I20260925 16:33:28.748379 8486 tablet_replica.cc:439] T a636caf550ff41389999542e067c9a4a P 9203d42b2ff045c386ade97cea8d1430: TxnStatusTablet state changed. Reason: New leader 9203d42b2ff045c386ade97cea8d1430. Latest consensus state: current_term: 1 leader_uuid: "9203d42b2ff045c386ade97cea8d1430" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948c869ed1fe452e8f64517c105e455d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41731 } } peers { permanent_uuid: "f649366115124f25bc74e77e773613ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } } peers { permanent_uuid: "9203d42b2ff045c386ade97cea8d1430" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36421 } } }
I20260925 16:33:28.748492 8486 tablet_replica.cc:442] T a636caf550ff41389999542e067c9a4a P 9203d42b2ff045c386ade97cea8d1430: This TxnStatusTablet replica's current role is: LEADER
I20260925 16:33:28.748721 8488 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 16:33:28.748868 8488 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 16:33:28.749583 7932 catalog_manager.cc:5495] T a636caf550ff41389999542e067c9a4a P 9203d42b2ff045c386ade97cea8d1430 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9203d42b2ff045c386ade97cea8d1430 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "9203d42b2ff045c386ade97cea8d1430" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948c869ed1fe452e8f64517c105e455d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f649366115124f25bc74e77e773613ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9203d42b2ff045c386ade97cea8d1430" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36421 } health_report { overall_health: HEALTHY } } }
I20260925 16:33:28.766278 8478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:28.762099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36421 (local address 127.0.0.1:58236)
0925 16:33:28.762346 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:28.762363 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:28.762459 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:28.762895 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:28.762899 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:28.762909 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 16:33:28.763021 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:28.763030 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:28.763753 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:28.763757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:28.764752 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:28.764763 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:28.765208 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:28.765212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:28.765302 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:28.765360 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 16:33:28.765968 (+ 608us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 16:33:28.765981 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 16:33:28.766055 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 16:33:28.766139 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134}
I20260925 16:33:28.766299 8491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:28.762189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58236 (local address 127.0.98.3:36421)
0925 16:33:28.762440 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:28.762446 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:28.762459 (+ 13us) server_negotiation.cc:407] Connection header received
0925 16:33:28.762658 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:28.762664 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:28.762746 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:28.762870 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:33:28.763133 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:28.763653 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:28.764879 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:28.765108 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:28.765183 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:28.765493 (+ 310us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:33:28.765879 (+ 386us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:33:28.765963 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:28.766075 (+ 112us) server_negotiation.cc:299] Negotiation successful
0925 16:33:28.766120 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":105,"threads_started":1}
I20260925 16:33:28.769224 8229 raft_consensus.cc:1261] T a636caf550ff41389999542e067c9a4a P f649366115124f25bc74e77e773613ad [term 1 FOLLOWER]: Refusing update from remote peer 9203d42b2ff045c386ade97cea8d1430: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:33:28.769524 8089 raft_consensus.cc:1261] T a636caf550ff41389999542e067c9a4a P 948c869ed1fe452e8f64517c105e455d [term 1 FOLLOWER]: Refusing update from remote peer 9203d42b2ff045c386ade97cea8d1430: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:33:28.769702 8486 consensus_queue.cc:1035] T a636caf550ff41389999542e067c9a4a P 9203d42b2ff045c386ade97cea8d1430 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f649366115124f25bc74e77e773613ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:33:28.769991 8451 consensus_queue.cc:1035] T a636caf550ff41389999542e067c9a4a P 9203d42b2ff045c386ade97cea8d1430 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948c869ed1fe452e8f64517c105e455d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:33:28.771191 8449 tablet_replica.cc:439] T a636caf550ff41389999542e067c9a4a P f649366115124f25bc74e77e773613ad: TxnStatusTablet state changed. Reason: New leader 9203d42b2ff045c386ade97cea8d1430. Latest consensus state: current_term: 1 leader_uuid: "9203d42b2ff045c386ade97cea8d1430" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948c869ed1fe452e8f64517c105e455d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41731 } } peers { permanent_uuid: "f649366115124f25bc74e77e773613ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } } peers { permanent_uuid: "9203d42b2ff045c386ade97cea8d1430" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36421 } } }
I20260925 16:33:28.771304 8449 tablet_replica.cc:442] T a636caf550ff41389999542e067c9a4a P f649366115124f25bc74e77e773613ad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 16:33:28.771659 8454 tablet_replica.cc:439] T a636caf550ff41389999542e067c9a4a P 948c869ed1fe452e8f64517c105e455d: TxnStatusTablet state changed. Reason: New leader 9203d42b2ff045c386ade97cea8d1430. Latest consensus state: current_term: 1 leader_uuid: "9203d42b2ff045c386ade97cea8d1430" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948c869ed1fe452e8f64517c105e455d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41731 } } peers { permanent_uuid: "f649366115124f25bc74e77e773613ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } } peers { permanent_uuid: "9203d42b2ff045c386ade97cea8d1430" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36421 } } }
I20260925 16:33:28.771776 8454 tablet_replica.cc:442] T a636caf550ff41389999542e067c9a4a P 948c869ed1fe452e8f64517c105e455d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 16:33:28.774947 8449 tablet_replica.cc:439] T a636caf550ff41389999542e067c9a4a P f649366115124f25bc74e77e773613ad: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9203d42b2ff045c386ade97cea8d1430" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948c869ed1fe452e8f64517c105e455d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41731 } } peers { permanent_uuid: "f649366115124f25bc74e77e773613ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } } peers { permanent_uuid: "9203d42b2ff045c386ade97cea8d1430" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36421 } } }
I20260925 16:33:28.775055 8449 tablet_replica.cc:442] T a636caf550ff41389999542e067c9a4a P f649366115124f25bc74e77e773613ad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 16:33:28.774633 8497 tablet_replica.cc:439] T a636caf550ff41389999542e067c9a4a P 9203d42b2ff045c386ade97cea8d1430: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9203d42b2ff045c386ade97cea8d1430" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948c869ed1fe452e8f64517c105e455d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41731 } } peers { permanent_uuid: "f649366115124f25bc74e77e773613ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } } peers { permanent_uuid: "9203d42b2ff045c386ade97cea8d1430" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36421 } } }
I20260925 16:33:28.775110 8497 tablet_replica.cc:442] T a636caf550ff41389999542e067c9a4a P 9203d42b2ff045c386ade97cea8d1430: This TxnStatusTablet replica's current role is: LEADER
I20260925 16:33:28.775283 8486 tablet_replica.cc:439] T a636caf550ff41389999542e067c9a4a P 9203d42b2ff045c386ade97cea8d1430: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9203d42b2ff045c386ade97cea8d1430" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948c869ed1fe452e8f64517c105e455d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41731 } } peers { permanent_uuid: "f649366115124f25bc74e77e773613ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } } peers { permanent_uuid: "9203d42b2ff045c386ade97cea8d1430" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36421 } } }
I20260925 16:33:28.775390 8486 tablet_replica.cc:442] T a636caf550ff41389999542e067c9a4a P 9203d42b2ff045c386ade97cea8d1430: This TxnStatusTablet replica's current role is: LEADER
I20260925 16:33:28.776139 8454 tablet_replica.cc:439] T a636caf550ff41389999542e067c9a4a P 948c869ed1fe452e8f64517c105e455d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9203d42b2ff045c386ade97cea8d1430" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948c869ed1fe452e8f64517c105e455d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41731 } } peers { permanent_uuid: "f649366115124f25bc74e77e773613ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } } peers { permanent_uuid: "9203d42b2ff045c386ade97cea8d1430" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36421 } } }
I20260925 16:33:28.776242 8454 tablet_replica.cc:442] T a636caf550ff41389999542e067c9a4a P 948c869ed1fe452e8f64517c105e455d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 16:33:28.794778 8491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:28.786814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48503 (local address 127.0.98.3:36421)
0925 16:33:28.786957 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:28.786961 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:28.790894 (+ 3933us) server_negotiation.cc:407] Connection header received
0925 16:33:28.791017 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:28.791020 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:28.791072 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:28.791151 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:33:28.791398 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:28.792164 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:28.793520 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:28.793861 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:28.793978 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:28.794263 (+ 285us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:33:28.794339 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:33:28.794402 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:28.794556 (+ 154us) server_negotiation.cc:299] Negotiation successful
0925 16:33:28.794633 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 16:33:28.794780 8503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:28.786735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36421 (local address 127.0.98.1:48503)
0925 16:33:28.790791 (+ 4056us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:28.790808 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:28.790917 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:28.791170 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:28.791174 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:28.791182 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 16:33:28.791288 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:28.791294 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:28.792275 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:28.792279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:28.793280 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:28.793291 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:28.793984 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:28.793988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:28.794083 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:28.794133 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 16:33:28.794439 (+ 306us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 16:33:28.794446 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 16:33:28.794529 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 16:33:28.794629 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3952,"thread_start_us":97,"threads_started":1}
I20260925 16:33:28.802127 8368 raft_consensus.cc:1261] T 8fe0c09b6acc48b2b397fc423637aea3 P 9203d42b2ff045c386ade97cea8d1430 [term 1 FOLLOWER]: Refusing update from remote peer 948c869ed1fe452e8f64517c105e455d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:33:28.802201 8229 raft_consensus.cc:1261] T 8fe0c09b6acc48b2b397fc423637aea3 P f649366115124f25bc74e77e773613ad [term 1 FOLLOWER]: Refusing update from remote peer 948c869ed1fe452e8f64517c105e455d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:33:28.802541 8450 consensus_queue.cc:1035] T 8fe0c09b6acc48b2b397fc423637aea3 P 948c869ed1fe452e8f64517c105e455d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9203d42b2ff045c386ade97cea8d1430" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:33:28.802758 8454 consensus_queue.cc:1035] T 8fe0c09b6acc48b2b397fc423637aea3 P 948c869ed1fe452e8f64517c105e455d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f649366115124f25bc74e77e773613ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:33:28.806443 8492 mvcc.cc:205] Tried to move back new op lower bound from 7333290020049674240 to 7333290018804375552. Current Snapshot: MvccSnapshot[applied={T|T < 7333290020049674240}]
I20260925 16:33:28.824057 8483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:28.820095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57937 (local address 127.0.98.1:41731)
0925 16:33:28.820301 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:28.820306 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:28.820424 (+ 118us) server_negotiation.cc:407] Connection header received
0925 16:33:28.820614 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:28.820618 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:28.820665 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:28.820712 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:33:28.821117 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:28.821828 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:28.822855 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:28.823090 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:28.823205 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:28.823540 (+ 335us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:33:28.823625 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:33:28.823674 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:28.823859 (+ 185us) server_negotiation.cc:299] Negotiation successful
0925 16:33:28.823913 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260925 16:33:28.824533 8507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:28.819708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41731 (local address 127.0.98.3:57937)
0925 16:33:28.820343 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:28.820356 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:28.820500 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:28.820855 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:28.820858 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:28.820866 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 16:33:28.820995 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:28.821002 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:28.821998 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:28.822001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:28.822737 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:28.822747 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:28.823204 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:28.823208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:28.823322 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:28.823370 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 16:33:28.823757 (+ 387us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 16:33:28.823763 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 16:33:28.824263 (+ 500us) client_negotiation.cc:240] Negotiation successful
0925 16:33:28.824374 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":106,"threads_started":1}
I20260925 16:33:28.845541 7931 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:33:28.845721 7931 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:33:28.847731 7931 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7bf7b1c2fbb04b7a96e4038edbe73e92: Sending DeleteTablet for 3 replicas of tablet 8fe0c09b6acc48b2b397fc423637aea3
I20260925 16:33:28.848289 8209 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fe0c09b6acc48b2b397fc423637aea3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:33:28 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58520
I20260925 16:33:28.848659 8510 tablet_replica.cc:330] T 8fe0c09b6acc48b2b397fc423637aea3 P f649366115124f25bc74e77e773613ad: stopping tablet replica
I20260925 16:33:28.848785 8510 raft_consensus.cc:2229] T 8fe0c09b6acc48b2b397fc423637aea3 P f649366115124f25bc74e77e773613ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:33:28.848779 8069 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fe0c09b6acc48b2b397fc423637aea3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:33:28 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51892
I20260925 16:33:28.848892 8510 raft_consensus.cc:2258] T 8fe0c09b6acc48b2b397fc423637aea3 P f649366115124f25bc74e77e773613ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:33:28.851899 8349 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fe0c09b6acc48b2b397fc423637aea3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:33:28 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58224
I20260925 16:33:28.852269 8513 tablet_replica.cc:330] T 8fe0c09b6acc48b2b397fc423637aea3 P 9203d42b2ff045c386ade97cea8d1430: stopping tablet replica
I20260925 16:33:28.852401 8513 raft_consensus.cc:2229] T 8fe0c09b6acc48b2b397fc423637aea3 P 9203d42b2ff045c386ade97cea8d1430 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:33:28.852490 8510 ts_tablet_manager.cc:1905] T 8fe0c09b6acc48b2b397fc423637aea3 P f649366115124f25bc74e77e773613ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:33:28.852510 8513 raft_consensus.cc:2258] T 8fe0c09b6acc48b2b397fc423637aea3 P 9203d42b2ff045c386ade97cea8d1430 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:33:28.853127 8513 ts_tablet_manager.cc:1905] T 8fe0c09b6acc48b2b397fc423637aea3 P 9203d42b2ff045c386ade97cea8d1430: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:33:28.854760 8511 tablet_replica.cc:330] T 8fe0c09b6acc48b2b397fc423637aea3 P 948c869ed1fe452e8f64517c105e455d: stopping tablet replica
I20260925 16:33:28.854885 8511 raft_consensus.cc:2229] T 8fe0c09b6acc48b2b397fc423637aea3 P 948c869ed1fe452e8f64517c105e455d [term 1 LEADER]: Raft consensus shutting down.
I20260925 16:33:28.855067 8511 raft_consensus.cc:2258] T 8fe0c09b6acc48b2b397fc423637aea3 P 948c869ed1fe452e8f64517c105e455d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:33:28.855620 8510 ts_tablet_manager.cc:1918] T 8fe0c09b6acc48b2b397fc423637aea3 P f649366115124f25bc74e77e773613ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 16:33:28.855685 8510 log.cc:1196] T 8fe0c09b6acc48b2b397fc423637aea3 P f649366115124f25bc74e77e773613ad: Deleting WAL directory at /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/ts-1/wal/wals/8fe0c09b6acc48b2b397fc423637aea3
I20260925 16:33:28.855935 8510 ts_tablet_manager.cc:1939] T 8fe0c09b6acc48b2b397fc423637aea3 P f649366115124f25bc74e77e773613ad: Deleting consensus metadata
I20260925 16:33:28.856333 8511 ts_tablet_manager.cc:1905] T 8fe0c09b6acc48b2b397fc423637aea3 P 948c869ed1fe452e8f64517c105e455d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:33:28.856796 7916 catalog_manager.cc:4841] TS f649366115124f25bc74e77e773613ad (127.0.98.2:40387): tablet 8fe0c09b6acc48b2b397fc423637aea3 (table test-table [id=be7a7c20356c4aa0b2c3aa9fe3c3c3f1]) successfully deleted
I20260925 16:33:28.858633 8513 ts_tablet_manager.cc:1918] T 8fe0c09b6acc48b2b397fc423637aea3 P 9203d42b2ff045c386ade97cea8d1430: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 16:33:28.858723 8513 log.cc:1196] T 8fe0c09b6acc48b2b397fc423637aea3 P 9203d42b2ff045c386ade97cea8d1430: Deleting WAL directory at /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/ts-2/wal/wals/8fe0c09b6acc48b2b397fc423637aea3
I20260925 16:33:28.859002 8513 ts_tablet_manager.cc:1939] T 8fe0c09b6acc48b2b397fc423637aea3 P 9203d42b2ff045c386ade97cea8d1430: Deleting consensus metadata
I20260925 16:33:28.859345 8511 ts_tablet_manager.cc:1918] T 8fe0c09b6acc48b2b397fc423637aea3 P 948c869ed1fe452e8f64517c105e455d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 16:33:28.859459 8511 log.cc:1196] T 8fe0c09b6acc48b2b397fc423637aea3 P 948c869ed1fe452e8f64517c105e455d: Deleting WAL directory at /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/ts-0/wal/wals/8fe0c09b6acc48b2b397fc423637aea3
I20260925 16:33:28.859805 8511 ts_tablet_manager.cc:1939] T 8fe0c09b6acc48b2b397fc423637aea3 P 948c869ed1fe452e8f64517c105e455d: Deleting consensus metadata
I20260925 16:33:28.859719 7918 catalog_manager.cc:4841] TS 9203d42b2ff045c386ade97cea8d1430 (127.0.98.3:36421): tablet 8fe0c09b6acc48b2b397fc423637aea3 (table test-table [id=be7a7c20356c4aa0b2c3aa9fe3c3c3f1]) successfully deleted
I20260925 16:33:28.860460 7917 catalog_manager.cc:4841] TS 948c869ed1fe452e8f64517c105e455d (127.0.98.1:41731): tablet 8fe0c09b6acc48b2b397fc423637aea3 (table test-table [id=be7a7c20356c4aa0b2c3aa9fe3c3c3f1]) successfully deleted
Sep 25 16:33:28 dist-test-slave-65pf krb5kdc[7887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 16:33:28 dist-test-slave-65pf krb5kdc[7887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:33:28 dist-test-slave-65pf krb5kdc[7887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 16:33:28.881536 7981 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:33:28.874747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36630 (local address 127.0.98.62:37535)
0925 16:33:28.874898 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:28.874904 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:28.875012 (+ 108us) server_negotiation.cc:407] Connection header received
0925 16:33:28.875134 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:28.875139 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:28.875198 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:28.875297 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:28.876067 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:28.876906 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:28.877762 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:28.878115 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:28.878226 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:28.881394 (+ 3168us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36630: BlockingRecv error: recv got EOF from 127.0.0.1:36630 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 25 16:33:28 dist-test-slave-65pf krb5kdc[7887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 16:33:28.892457 7981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:28.883203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36636 (local address 127.0.98.62:37535)
0925 16:33:28.883336 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:28.883340 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:28.883538 (+ 198us) server_negotiation.cc:407] Connection header received
0925 16:33:28.883666 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:28.883668 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:28.883710 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:28.883783 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:28.884530 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:28.885057 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:28.886259 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:28.886521 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:28.886677 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:28.889255 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:28.889264 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:28.889266 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:28.889289 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:28.890794 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:28.891210 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:28.891215 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:28.891217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:28.891298 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:28.891592 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:28.891596 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:28.891598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:28.891777 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:28.891912 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:28.892162 (+ 250us) server_negotiation.cc:299] Negotiation successful
0925 16:33:28.892279 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 16:33:28.895198 7931 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36636:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 16:33:28.895713 7931 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:33:28.900462 8069 tablet_service.cc:1467] Processing CreateTablet for tablet 13bc462be22c4ae18ecc8149387d810f (DEFAULT_TABLE table=test-table [id=035718ba546e4a669d52a4f171127e1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:33:28.900570 8209 tablet_service.cc:1467] Processing CreateTablet for tablet 13bc462be22c4ae18ecc8149387d810f (DEFAULT_TABLE table=test-table [id=035718ba546e4a669d52a4f171127e1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:33:28.900822 8069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13bc462be22c4ae18ecc8149387d810f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:33:28.900904 8209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13bc462be22c4ae18ecc8149387d810f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:33:28.902980 8349 tablet_service.cc:1467] Processing CreateTablet for tablet 13bc462be22c4ae18ecc8149387d810f (DEFAULT_TABLE table=test-table [id=035718ba546e4a669d52a4f171127e1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:33:28.903285 8349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13bc462be22c4ae18ecc8149387d810f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:33:28.903342 8445 tablet_bootstrap.cc:492] T 13bc462be22c4ae18ecc8149387d810f P 948c869ed1fe452e8f64517c105e455d: Bootstrap starting.
I20260925 16:33:28.904172 8445 tablet_bootstrap.cc:654] T 13bc462be22c4ae18ecc8149387d810f P 948c869ed1fe452e8f64517c105e455d: Neither blocks nor log segments found. Creating new log.
I20260925 16:33:28.905359 8445 tablet_bootstrap.cc:492] T 13bc462be22c4ae18ecc8149387d810f P 948c869ed1fe452e8f64517c105e455d: No bootstrap required, opened a new log
I20260925 16:33:28.905438 8444 tablet_bootstrap.cc:492] T 13bc462be22c4ae18ecc8149387d810f P f649366115124f25bc74e77e773613ad: Bootstrap starting.
I20260925 16:33:28.905439 8445 ts_tablet_manager.cc:1397] T 13bc462be22c4ae18ecc8149387d810f P 948c869ed1fe452e8f64517c105e455d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 16:33:28.905857 8445 raft_consensus.cc:348] T 13bc462be22c4ae18ecc8149387d810f P 948c869ed1fe452e8f64517c105e455d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948c869ed1fe452e8f64517c105e455d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41731 } } peers { permanent_uuid: "f649366115124f25bc74e77e773613ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } } peers { permanent_uuid: "9203d42b2ff045c386ade97cea8d1430" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36421 } }
I20260925 16:33:28.905972 8445 raft_consensus.cc:374] T 13bc462be22c4ae18ecc8149387d810f P 948c869ed1fe452e8f64517c105e455d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:33:28.906002 8445 raft_consensus.cc:729] T 13bc462be22c4ae18ecc8149387d810f P 948c869ed1fe452e8f64517c105e455d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948c869ed1fe452e8f64517c105e455d, State: Initialized, Role: FOLLOWER
I20260925 16:33:28.906124 8445 consensus_queue.cc:260] T 13bc462be22c4ae18ecc8149387d810f P 948c869ed1fe452e8f64517c105e455d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948c869ed1fe452e8f64517c105e455d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41731 } } peers { permanent_uuid: "f649366115124f25bc74e77e773613ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } } peers { permanent_uuid: "9203d42b2ff045c386ade97cea8d1430" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36421 } }
I20260925 16:33:28.906349 8444 tablet_bootstrap.cc:654] T 13bc462be22c4ae18ecc8149387d810f P f649366115124f25bc74e77e773613ad: Neither blocks nor log segments found. Creating new log.
I20260925 16:33:28.906399 8445 ts_tablet_manager.cc:1428] T 13bc462be22c4ae18ecc8149387d810f P 948c869ed1fe452e8f64517c105e455d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 16:33:28.907024 8443 tablet_bootstrap.cc:492] T 13bc462be22c4ae18ecc8149387d810f P 9203d42b2ff045c386ade97cea8d1430: Bootstrap starting.
I20260925 16:33:28.908013 8443 tablet_bootstrap.cc:654] T 13bc462be22c4ae18ecc8149387d810f P 9203d42b2ff045c386ade97cea8d1430: Neither blocks nor log segments found. Creating new log.
I20260925 16:33:28.909575 8443 tablet_bootstrap.cc:492] T 13bc462be22c4ae18ecc8149387d810f P 9203d42b2ff045c386ade97cea8d1430: No bootstrap required, opened a new log
I20260925 16:33:28.909655 8443 ts_tablet_manager.cc:1397] T 13bc462be22c4ae18ecc8149387d810f P 9203d42b2ff045c386ade97cea8d1430: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 16:33:28.910001 8444 tablet_bootstrap.cc:492] T 13bc462be22c4ae18ecc8149387d810f P f649366115124f25bc74e77e773613ad: No bootstrap required, opened a new log
I20260925 16:33:28.910075 8444 ts_tablet_manager.cc:1397] T 13bc462be22c4ae18ecc8149387d810f P f649366115124f25bc74e77e773613ad: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260925 16:33:28.910302 8443 raft_consensus.cc:348] T 13bc462be22c4ae18ecc8149387d810f P 9203d42b2ff045c386ade97cea8d1430 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948c869ed1fe452e8f64517c105e455d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41731 } } peers { permanent_uuid: "f649366115124f25bc74e77e773613ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } } peers { permanent_uuid: "9203d42b2ff045c386ade97cea8d1430" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36421 } }
I20260925 16:33:28.910423 8443 raft_consensus.cc:374] T 13bc462be22c4ae18ecc8149387d810f P 9203d42b2ff045c386ade97cea8d1430 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:33:28.910466 8443 raft_consensus.cc:729] T 13bc462be22c4ae18ecc8149387d810f P 9203d42b2ff045c386ade97cea8d1430 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9203d42b2ff045c386ade97cea8d1430, State: Initialized, Role: FOLLOWER
I20260925 16:33:28.910645 8443 consensus_queue.cc:260] T 13bc462be22c4ae18ecc8149387d810f P 9203d42b2ff045c386ade97cea8d1430 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948c869ed1fe452e8f64517c105e455d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41731 } } peers { permanent_uuid: "f649366115124f25bc74e77e773613ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } } peers { permanent_uuid: "9203d42b2ff045c386ade97cea8d1430" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36421 } }
I20260925 16:33:28.910873 8444 raft_consensus.cc:348] T 13bc462be22c4ae18ecc8149387d810f P f649366115124f25bc74e77e773613ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948c869ed1fe452e8f64517c105e455d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41731 } } peers { permanent_uuid: "f649366115124f25bc74e77e773613ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } } peers { permanent_uuid: "9203d42b2ff045c386ade97cea8d1430" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36421 } }
I20260925 16:33:28.910936 8443 ts_tablet_manager.cc:1428] T 13bc462be22c4ae18ecc8149387d810f P 9203d42b2ff045c386ade97cea8d1430: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 16:33:28.910998 8444 raft_consensus.cc:374] T 13bc462be22c4ae18ecc8149387d810f P f649366115124f25bc74e77e773613ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:33:28.911041 8444 raft_consensus.cc:729] T 13bc462be22c4ae18ecc8149387d810f P f649366115124f25bc74e77e773613ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f649366115124f25bc74e77e773613ad, State: Initialized, Role: FOLLOWER
I20260925 16:33:28.911175 8444 consensus_queue.cc:260] T 13bc462be22c4ae18ecc8149387d810f P f649366115124f25bc74e77e773613ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948c869ed1fe452e8f64517c105e455d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41731 } } peers { permanent_uuid: "f649366115124f25bc74e77e773613ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } } peers { permanent_uuid: "9203d42b2ff045c386ade97cea8d1430" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36421 } }
I20260925 16:33:28.912935 8444 ts_tablet_manager.cc:1428] T 13bc462be22c4ae18ecc8149387d810f P f649366115124f25bc74e77e773613ad: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 16:33:28.918565 8449 raft_consensus.cc:482] T 13bc462be22c4ae18ecc8149387d810f P f649366115124f25bc74e77e773613ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:33:28.918725 8449 raft_consensus.cc:504] T 13bc462be22c4ae18ecc8149387d810f P f649366115124f25bc74e77e773613ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948c869ed1fe452e8f64517c105e455d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41731 } } peers { permanent_uuid: "f649366115124f25bc74e77e773613ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } } peers { permanent_uuid: "9203d42b2ff045c386ade97cea8d1430" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36421 } }
I20260925 16:33:28.919817 8449 leader_election.cc:288] T 13bc462be22c4ae18ecc8149387d810f P f649366115124f25bc74e77e773613ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948c869ed1fe452e8f64517c105e455d (127.0.98.1:41731), 9203d42b2ff045c386ade97cea8d1430 (127.0.98.3:36421)
I20260925 16:33:28.924134 8529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:28.920088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36421 (local address 127.0.98.2:54527)
0925 16:33:28.920421 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:28.920435 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:28.920560 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:28.920926 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:28.920929 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:28.920947 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:33:28.921060 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:28.921072 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:28.922038 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:28.922040 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:28.922957 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:28.922965 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:28.923659 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:28.923663 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:28.923771 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:28.923800 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 16:33:28.923889 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 16:33:28.923970 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":2,"thread_start_us":86,"threads_started":1}
I20260925 16:33:28.924121 8491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:28.920168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54527 (local address 127.0.98.3:36421)
0925 16:33:28.920280 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:28.920285 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:28.920499 (+ 214us) server_negotiation.cc:407] Connection header received
0925 16:33:28.920693 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:28.920697 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:28.920740 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:28.920806 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:33:28.921186 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:28.921895 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:28.923079 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:28.923533 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:28.923660 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:28.923729 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:28.923953 (+ 224us) server_negotiation.cc:299] Negotiation successful
0925 16:33:28.924022 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 16:33:28.924499 8368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13bc462be22c4ae18ecc8149387d810f" candidate_uuid: "f649366115124f25bc74e77e773613ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9203d42b2ff045c386ade97cea8d1430" is_pre_election: true
I20260925 16:33:28.924644 8368 raft_consensus.cc:2454] T 13bc462be22c4ae18ecc8149387d810f P 9203d42b2ff045c386ade97cea8d1430 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f649366115124f25bc74e77e773613ad in term 0.
I20260925 16:33:28.925002 8162 leader_election.cc:302] T 13bc462be22c4ae18ecc8149387d810f P f649366115124f25bc74e77e773613ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9203d42b2ff045c386ade97cea8d1430, f649366115124f25bc74e77e773613ad; no voters:
I20260925 16:33:28.925277 8449 raft_consensus.cc:2781] T 13bc462be22c4ae18ecc8149387d810f P f649366115124f25bc74e77e773613ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:33:28.925307 8483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:28.920366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52557 (local address 127.0.98.1:41731)
0925 16:33:28.920502 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:28.920509 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:28.920525 (+ 16us) server_negotiation.cc:407] Connection header received
0925 16:33:28.920585 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:28.920590 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:28.920643 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:28.920711 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:33:28.920984 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:28.921782 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:28.924090 (+ 2308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:28.924698 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:28.924779 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:28.924836 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:28.925112 (+ 276us) server_negotiation.cc:299] Negotiation successful
0925 16:33:28.925188 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 16:33:28.925346 8449 raft_consensus.cc:482] T 13bc462be22c4ae18ecc8149387d810f P f649366115124f25bc74e77e773613ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:33:28.925400 8449 raft_consensus.cc:3037] T 13bc462be22c4ae18ecc8149387d810f P f649366115124f25bc74e77e773613ad [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:33:28.925639 8528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:28.920002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41731 (local address 127.0.98.2:52557)
0925 16:33:28.920361 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:28.920377 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:28.920471 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:28.920729 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:28.920732 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:28.920745 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:33:28.920887 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:28.920895 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:28.921898 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:28.921902 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:28.923955 (+ 2053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:28.923965 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:28.924878 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:28.924882 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:28.924977 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:28.925004 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 16:33:28.925084 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 16:33:28.925237 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"mutex_wait_us":25,"thread_start_us":112,"threads_started":1}
I20260925 16:33:28.926067 8089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13bc462be22c4ae18ecc8149387d810f" candidate_uuid: "f649366115124f25bc74e77e773613ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948c869ed1fe452e8f64517c105e455d" is_pre_election: true
I20260925 16:33:28.926221 8089 raft_consensus.cc:2454] T 13bc462be22c4ae18ecc8149387d810f P 948c869ed1fe452e8f64517c105e455d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f649366115124f25bc74e77e773613ad in term 0.
I20260925 16:33:28.926680 8449 raft_consensus.cc:504] T 13bc462be22c4ae18ecc8149387d810f P f649366115124f25bc74e77e773613ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948c869ed1fe452e8f64517c105e455d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41731 } } peers { permanent_uuid: "f649366115124f25bc74e77e773613ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } } peers { permanent_uuid: "9203d42b2ff045c386ade97cea8d1430" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36421 } }
I20260925 16:33:28.927016 8449 leader_election.cc:288] T 13bc462be22c4ae18ecc8149387d810f P f649366115124f25bc74e77e773613ad [CANDIDATE]: Term 1 election: Requested vote from peers 948c869ed1fe452e8f64517c105e455d (127.0.98.1:41731), 9203d42b2ff045c386ade97cea8d1430 (127.0.98.3:36421)
I20260925 16:33:28.927361 8368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13bc462be22c4ae18ecc8149387d810f" candidate_uuid: "f649366115124f25bc74e77e773613ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9203d42b2ff045c386ade97cea8d1430"
I20260925 16:33:28.927426 8089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13bc462be22c4ae18ecc8149387d810f" candidate_uuid: "f649366115124f25bc74e77e773613ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948c869ed1fe452e8f64517c105e455d"
I20260925 16:33:28.927477 8368 raft_consensus.cc:3037] T 13bc462be22c4ae18ecc8149387d810f P 9203d42b2ff045c386ade97cea8d1430 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:33:28.927541 8089 raft_consensus.cc:3037] T 13bc462be22c4ae18ecc8149387d810f P 948c869ed1fe452e8f64517c105e455d [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:33:28.928562 8089 raft_consensus.cc:2454] T 13bc462be22c4ae18ecc8149387d810f P 948c869ed1fe452e8f64517c105e455d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f649366115124f25bc74e77e773613ad in term 1.
I20260925 16:33:28.928562 8368 raft_consensus.cc:2454] T 13bc462be22c4ae18ecc8149387d810f P 9203d42b2ff045c386ade97cea8d1430 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f649366115124f25bc74e77e773613ad in term 1.
I20260925 16:33:28.928907 8161 leader_election.cc:302] T 13bc462be22c4ae18ecc8149387d810f P f649366115124f25bc74e77e773613ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 948c869ed1fe452e8f64517c105e455d, f649366115124f25bc74e77e773613ad; no voters:
I20260925 16:33:28.929332 8449 raft_consensus.cc:2781] T 13bc462be22c4ae18ecc8149387d810f P f649366115124f25bc74e77e773613ad [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:33:28.929557 8449 raft_consensus.cc:686] T 13bc462be22c4ae18ecc8149387d810f P f649366115124f25bc74e77e773613ad [term 1 LEADER]: Becoming Leader. State: Replica: f649366115124f25bc74e77e773613ad, State: Running, Role: LEADER
I20260925 16:33:28.929796 8449 consensus_queue.cc:237] T 13bc462be22c4ae18ecc8149387d810f P f649366115124f25bc74e77e773613ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948c869ed1fe452e8f64517c105e455d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41731 } } peers { permanent_uuid: "f649366115124f25bc74e77e773613ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } } peers { permanent_uuid: "9203d42b2ff045c386ade97cea8d1430" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36421 } }
I20260925 16:33:28.932108 7931 catalog_manager.cc:5495] T 13bc462be22c4ae18ecc8149387d810f P f649366115124f25bc74e77e773613ad reported cstate change: term changed from 0 to 1, leader changed from <none> to f649366115124f25bc74e77e773613ad (127.0.98.2). New cstate: current_term: 1 leader_uuid: "f649366115124f25bc74e77e773613ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948c869ed1fe452e8f64517c105e455d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f649366115124f25bc74e77e773613ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9203d42b2ff045c386ade97cea8d1430" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36421 } health_report { overall_health: UNKNOWN } } }
I20260925 16:33:28.944813 8484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:28.941215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58526 (local address 127.0.98.2:40387)
0925 16:33:28.941335 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:28.941339 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:28.941353 (+ 14us) server_negotiation.cc:407] Connection header received
0925 16:33:28.941522 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:28.941525 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:28.941580 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:28.941689 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:33:28.941920 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:28.942423 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:28.943337 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:28.943554 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:28.943660 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:28.943971 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:33:28.944409 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:33:28.944535 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:28.944597 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 16:33:28.944651 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 16:33:28.947468 8368 raft_consensus.cc:1261] T 13bc462be22c4ae18ecc8149387d810f P 9203d42b2ff045c386ade97cea8d1430 [term 1 FOLLOWER]: Refusing update from remote peer f649366115124f25bc74e77e773613ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:33:28.947492 8089 raft_consensus.cc:1261] T 13bc462be22c4ae18ecc8149387d810f P 948c869ed1fe452e8f64517c105e455d [term 1 FOLLOWER]: Refusing update from remote peer f649366115124f25bc74e77e773613ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:33:28.947988 8530 consensus_queue.cc:1035] T 13bc462be22c4ae18ecc8149387d810f P f649366115124f25bc74e77e773613ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9203d42b2ff045c386ade97cea8d1430" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:33:28.948175 8449 consensus_queue.cc:1035] T 13bc462be22c4ae18ecc8149387d810f P f649366115124f25bc74e77e773613ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "948c869ed1fe452e8f64517c105e455d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:33:28.958854 7931 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36636:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:33:28.958994 7931 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36636:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:33:28.961066 7931 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7bf7b1c2fbb04b7a96e4038edbe73e92: Sending DeleteTablet for 3 replicas of tablet 13bc462be22c4ae18ecc8149387d810f
I20260925 16:33:28.961647 8069 tablet_service.cc:1514] Processing DeleteTablet for tablet 13bc462be22c4ae18ecc8149387d810f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:33:28 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51892
I20260925 16:33:28.961879 8209 tablet_service.cc:1514] Processing DeleteTablet for tablet 13bc462be22c4ae18ecc8149387d810f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:33:28 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58520
I20260925 16:33:28.962067 8511 tablet_replica.cc:330] T 13bc462be22c4ae18ecc8149387d810f P 948c869ed1fe452e8f64517c105e455d: stopping tablet replica
I20260925 16:33:28.962193 8511 raft_consensus.cc:2229] T 13bc462be22c4ae18ecc8149387d810f P 948c869ed1fe452e8f64517c105e455d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:33:28.962359 8511 raft_consensus.cc:2258] T 13bc462be22c4ae18ecc8149387d810f P 948c869ed1fe452e8f64517c105e455d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:33:28.962532 8510 tablet_replica.cc:330] T 13bc462be22c4ae18ecc8149387d810f P f649366115124f25bc74e77e773613ad: stopping tablet replica
I20260925 16:33:28.962692 8510 raft_consensus.cc:2229] T 13bc462be22c4ae18ecc8149387d810f P f649366115124f25bc74e77e773613ad [term 1 LEADER]: Raft consensus shutting down.
I20260925 16:33:28.962661 7931 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36636:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 16:33:28.963028 8510 raft_consensus.cc:2258] T 13bc462be22c4ae18ecc8149387d810f P f649366115124f25bc74e77e773613ad [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 16:33:28.963070 7931 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:33:28.963616 8349 tablet_service.cc:1514] Processing DeleteTablet for tablet 13bc462be22c4ae18ecc8149387d810f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:33:28 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58224
I20260925 16:33:28.963811 8513 tablet_replica.cc:330] T 13bc462be22c4ae18ecc8149387d810f P 9203d42b2ff045c386ade97cea8d1430: stopping tablet replica
I20260925 16:33:28.963927 8513 raft_consensus.cc:2229] T 13bc462be22c4ae18ecc8149387d810f P 9203d42b2ff045c386ade97cea8d1430 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:33:28.964077 8513 raft_consensus.cc:2258] T 13bc462be22c4ae18ecc8149387d810f P 9203d42b2ff045c386ade97cea8d1430 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:33:28.964360 8511 ts_tablet_manager.cc:1905] T 13bc462be22c4ae18ecc8149387d810f P 948c869ed1fe452e8f64517c105e455d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:33:28.964753 8510 ts_tablet_manager.cc:1905] T 13bc462be22c4ae18ecc8149387d810f P f649366115124f25bc74e77e773613ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:33:28.965373 8513 ts_tablet_manager.cc:1905] T 13bc462be22c4ae18ecc8149387d810f P 9203d42b2ff045c386ade97cea8d1430: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:33:28.966988 8511 ts_tablet_manager.cc:1918] T 13bc462be22c4ae18ecc8149387d810f P 948c869ed1fe452e8f64517c105e455d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:33:28.967072 8511 log.cc:1196] T 13bc462be22c4ae18ecc8149387d810f P 948c869ed1fe452e8f64517c105e455d: Deleting WAL directory at /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/ts-0/wal/wals/13bc462be22c4ae18ecc8149387d810f
I20260925 16:33:28.967332 8511 ts_tablet_manager.cc:1939] T 13bc462be22c4ae18ecc8149387d810f P 948c869ed1fe452e8f64517c105e455d: Deleting consensus metadata
I20260925 16:33:28.967413 8513 ts_tablet_manager.cc:1918] T 13bc462be22c4ae18ecc8149387d810f P 9203d42b2ff045c386ade97cea8d1430: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:33:28.967521 8513 log.cc:1196] T 13bc462be22c4ae18ecc8149387d810f P 9203d42b2ff045c386ade97cea8d1430: Deleting WAL directory at /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/ts-2/wal/wals/13bc462be22c4ae18ecc8149387d810f
I20260925 16:33:28.967759 8513 ts_tablet_manager.cc:1939] T 13bc462be22c4ae18ecc8149387d810f P 9203d42b2ff045c386ade97cea8d1430: Deleting consensus metadata
I20260925 16:33:28.968029 7917 catalog_manager.cc:4841] TS 948c869ed1fe452e8f64517c105e455d (127.0.98.1:41731): tablet 13bc462be22c4ae18ecc8149387d810f (table test-table [id=035718ba546e4a669d52a4f171127e1e]) successfully deleted
I20260925 16:33:28.968437 7918 catalog_manager.cc:4841] TS 9203d42b2ff045c386ade97cea8d1430 (127.0.98.3:36421): tablet 13bc462be22c4ae18ecc8149387d810f (table test-table [id=035718ba546e4a669d52a4f171127e1e]) successfully deleted
I20260925 16:33:28.970722 8510 ts_tablet_manager.cc:1918] T 13bc462be22c4ae18ecc8149387d810f P f649366115124f25bc74e77e773613ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:33:28.970839 8510 log.cc:1196] T 13bc462be22c4ae18ecc8149387d810f P f649366115124f25bc74e77e773613ad: Deleting WAL directory at /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/ts-1/wal/wals/13bc462be22c4ae18ecc8149387d810f
I20260925 16:33:28.971122 8510 ts_tablet_manager.cc:1939] T 13bc462be22c4ae18ecc8149387d810f P f649366115124f25bc74e77e773613ad: Deleting consensus metadata
I20260925 16:33:28.971844 8069 tablet_service.cc:1467] Processing CreateTablet for tablet 4bce3c53b5474b77b60334022b4a28fa (DEFAULT_TABLE table=test-table [id=6877bcd387564cfcad0368c7ddbcfb59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:33:28.972127 8069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bce3c53b5474b77b60334022b4a28fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:33:28.972128 8349 tablet_service.cc:1467] Processing CreateTablet for tablet 4bce3c53b5474b77b60334022b4a28fa (DEFAULT_TABLE table=test-table [id=6877bcd387564cfcad0368c7ddbcfb59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:33:28.972368 8349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bce3c53b5474b77b60334022b4a28fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:33:28.974108 8445 tablet_bootstrap.cc:492] T 4bce3c53b5474b77b60334022b4a28fa P 948c869ed1fe452e8f64517c105e455d: Bootstrap starting.
I20260925 16:33:28.974349 8209 tablet_service.cc:1467] Processing CreateTablet for tablet 4bce3c53b5474b77b60334022b4a28fa (DEFAULT_TABLE table=test-table [id=6877bcd387564cfcad0368c7ddbcfb59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:33:28.974567 8209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bce3c53b5474b77b60334022b4a28fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:33:28.974938 8445 tablet_bootstrap.cc:654] T 4bce3c53b5474b77b60334022b4a28fa P 948c869ed1fe452e8f64517c105e455d: Neither blocks nor log segments found. Creating new log.
I20260925 16:33:28.975975 8445 tablet_bootstrap.cc:492] T 4bce3c53b5474b77b60334022b4a28fa P 948c869ed1fe452e8f64517c105e455d: No bootstrap required, opened a new log
I20260925 16:33:28.976060 8445 ts_tablet_manager.cc:1397] T 4bce3c53b5474b77b60334022b4a28fa P 948c869ed1fe452e8f64517c105e455d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 16:33:28.976428 8445 raft_consensus.cc:348] T 4bce3c53b5474b77b60334022b4a28fa P 948c869ed1fe452e8f64517c105e455d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948c869ed1fe452e8f64517c105e455d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41731 } } peers { permanent_uuid: "f649366115124f25bc74e77e773613ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } } peers { permanent_uuid: "9203d42b2ff045c386ade97cea8d1430" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36421 } }
I20260925 16:33:28.976539 8445 raft_consensus.cc:374] T 4bce3c53b5474b77b60334022b4a28fa P 948c869ed1fe452e8f64517c105e455d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:33:28.976569 8445 raft_consensus.cc:729] T 4bce3c53b5474b77b60334022b4a28fa P 948c869ed1fe452e8f64517c105e455d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948c869ed1fe452e8f64517c105e455d, State: Initialized, Role: FOLLOWER
I20260925 16:33:28.976686 8445 consensus_queue.cc:260] T 4bce3c53b5474b77b60334022b4a28fa P 948c869ed1fe452e8f64517c105e455d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948c869ed1fe452e8f64517c105e455d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41731 } } peers { permanent_uuid: "f649366115124f25bc74e77e773613ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } } peers { permanent_uuid: "9203d42b2ff045c386ade97cea8d1430" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36421 } }
I20260925 16:33:28.976801 8444 tablet_bootstrap.cc:492] T 4bce3c53b5474b77b60334022b4a28fa P f649366115124f25bc74e77e773613ad: Bootstrap starting.
I20260925 16:33:28.976871 8443 tablet_bootstrap.cc:492] T 4bce3c53b5474b77b60334022b4a28fa P 9203d42b2ff045c386ade97cea8d1430: Bootstrap starting.
I20260925 16:33:28.976940 8445 ts_tablet_manager.cc:1428] T 4bce3c53b5474b77b60334022b4a28fa P 948c869ed1fe452e8f64517c105e455d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 16:33:28.977662 8443 tablet_bootstrap.cc:654] T 4bce3c53b5474b77b60334022b4a28fa P 9203d42b2ff045c386ade97cea8d1430: Neither blocks nor log segments found. Creating new log.
I20260925 16:33:28.977662 8444 tablet_bootstrap.cc:654] T 4bce3c53b5474b77b60334022b4a28fa P f649366115124f25bc74e77e773613ad: Neither blocks nor log segments found. Creating new log.
I20260925 16:33:28.978928 8443 tablet_bootstrap.cc:492] T 4bce3c53b5474b77b60334022b4a28fa P 9203d42b2ff045c386ade97cea8d1430: No bootstrap required, opened a new log
I20260925 16:33:28.979027 8443 ts_tablet_manager.cc:1397] T 4bce3c53b5474b77b60334022b4a28fa P 9203d42b2ff045c386ade97cea8d1430: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 16:33:28.979461 8443 raft_consensus.cc:348] T 4bce3c53b5474b77b60334022b4a28fa P 9203d42b2ff045c386ade97cea8d1430 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948c869ed1fe452e8f64517c105e455d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41731 } } peers { permanent_uuid: "f649366115124f25bc74e77e773613ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } } peers { permanent_uuid: "9203d42b2ff045c386ade97cea8d1430" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36421 } }
I20260925 16:33:28.979614 8443 raft_consensus.cc:374] T 4bce3c53b5474b77b60334022b4a28fa P 9203d42b2ff045c386ade97cea8d1430 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:33:28.979648 8443 raft_consensus.cc:729] T 4bce3c53b5474b77b60334022b4a28fa P 9203d42b2ff045c386ade97cea8d1430 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9203d42b2ff045c386ade97cea8d1430, State: Initialized, Role: FOLLOWER
I20260925 16:33:28.979785 8443 consensus_queue.cc:260] T 4bce3c53b5474b77b60334022b4a28fa P 9203d42b2ff045c386ade97cea8d1430 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948c869ed1fe452e8f64517c105e455d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41731 } } peers { permanent_uuid: "f649366115124f25bc74e77e773613ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } } peers { permanent_uuid: "9203d42b2ff045c386ade97cea8d1430" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36421 } }
I20260925 16:33:28.980055 8443 ts_tablet_manager.cc:1428] T 4bce3c53b5474b77b60334022b4a28fa P 9203d42b2ff045c386ade97cea8d1430: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 16:33:28.980526 7916 catalog_manager.cc:4841] TS f649366115124f25bc74e77e773613ad (127.0.98.2:40387): tablet 13bc462be22c4ae18ecc8149387d810f (table test-table [id=035718ba546e4a669d52a4f171127e1e]) successfully deleted
I20260925 16:33:28.981616 8444 tablet_bootstrap.cc:492] T 4bce3c53b5474b77b60334022b4a28fa P f649366115124f25bc74e77e773613ad: No bootstrap required, opened a new log
I20260925 16:33:28.981686 8444 ts_tablet_manager.cc:1397] T 4bce3c53b5474b77b60334022b4a28fa P f649366115124f25bc74e77e773613ad: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260925 16:33:28.982100 8444 raft_consensus.cc:348] T 4bce3c53b5474b77b60334022b4a28fa P f649366115124f25bc74e77e773613ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948c869ed1fe452e8f64517c105e455d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41731 } } peers { permanent_uuid: "f649366115124f25bc74e77e773613ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } } peers { permanent_uuid: "9203d42b2ff045c386ade97cea8d1430" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36421 } }
I20260925 16:33:28.982187 8444 raft_consensus.cc:374] T 4bce3c53b5474b77b60334022b4a28fa P f649366115124f25bc74e77e773613ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:33:28.982209 8444 raft_consensus.cc:729] T 4bce3c53b5474b77b60334022b4a28fa P f649366115124f25bc74e77e773613ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f649366115124f25bc74e77e773613ad, State: Initialized, Role: FOLLOWER
I20260925 16:33:28.982331 8444 consensus_queue.cc:260] T 4bce3c53b5474b77b60334022b4a28fa P f649366115124f25bc74e77e773613ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948c869ed1fe452e8f64517c105e455d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41731 } } peers { permanent_uuid: "f649366115124f25bc74e77e773613ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } } peers { permanent_uuid: "9203d42b2ff045c386ade97cea8d1430" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36421 } }
I20260925 16:33:28.982643 8444 ts_tablet_manager.cc:1428] T 4bce3c53b5474b77b60334022b4a28fa P f649366115124f25bc74e77e773613ad: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 16:33:29.130813 8454 raft_consensus.cc:482] T 4bce3c53b5474b77b60334022b4a28fa P 948c869ed1fe452e8f64517c105e455d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:33:29.130967 8454 raft_consensus.cc:504] T 4bce3c53b5474b77b60334022b4a28fa P 948c869ed1fe452e8f64517c105e455d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948c869ed1fe452e8f64517c105e455d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41731 } } peers { permanent_uuid: "f649366115124f25bc74e77e773613ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } } peers { permanent_uuid: "9203d42b2ff045c386ade97cea8d1430" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36421 } }
I20260925 16:33:29.131464 8454 leader_election.cc:288] T 4bce3c53b5474b77b60334022b4a28fa P 948c869ed1fe452e8f64517c105e455d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f649366115124f25bc74e77e773613ad (127.0.98.2:40387), 9203d42b2ff045c386ade97cea8d1430 (127.0.98.3:36421)
I20260925 16:33:29.131891 8368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bce3c53b5474b77b60334022b4a28fa" candidate_uuid: "948c869ed1fe452e8f64517c105e455d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9203d42b2ff045c386ade97cea8d1430" is_pre_election: true
I20260925 16:33:29.131912 8229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bce3c53b5474b77b60334022b4a28fa" candidate_uuid: "948c869ed1fe452e8f64517c105e455d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f649366115124f25bc74e77e773613ad" is_pre_election: true
I20260925 16:33:29.132056 8368 raft_consensus.cc:2454] T 4bce3c53b5474b77b60334022b4a28fa P 9203d42b2ff045c386ade97cea8d1430 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948c869ed1fe452e8f64517c105e455d in term 0.
I20260925 16:33:29.132061 8229 raft_consensus.cc:2454] T 4bce3c53b5474b77b60334022b4a28fa P f649366115124f25bc74e77e773613ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948c869ed1fe452e8f64517c105e455d in term 0.
I20260925 16:33:29.132387 8022 leader_election.cc:302] T 4bce3c53b5474b77b60334022b4a28fa P 948c869ed1fe452e8f64517c105e455d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9203d42b2ff045c386ade97cea8d1430, 948c869ed1fe452e8f64517c105e455d; no voters:
I20260925 16:33:29.132629 8454 raft_consensus.cc:2781] T 4bce3c53b5474b77b60334022b4a28fa P 948c869ed1fe452e8f64517c105e455d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:33:29.132711 8454 raft_consensus.cc:482] T 4bce3c53b5474b77b60334022b4a28fa P 948c869ed1fe452e8f64517c105e455d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:33:29.132766 8454 raft_consensus.cc:3037] T 4bce3c53b5474b77b60334022b4a28fa P 948c869ed1fe452e8f64517c105e455d [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:33:29.133827 8454 raft_consensus.cc:504] T 4bce3c53b5474b77b60334022b4a28fa P 948c869ed1fe452e8f64517c105e455d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948c869ed1fe452e8f64517c105e455d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41731 } } peers { permanent_uuid: "f649366115124f25bc74e77e773613ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } } peers { permanent_uuid: "9203d42b2ff045c386ade97cea8d1430" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36421 } }
I20260925 16:33:29.134152 8454 leader_election.cc:288] T 4bce3c53b5474b77b60334022b4a28fa P 948c869ed1fe452e8f64517c105e455d [CANDIDATE]: Term 1 election: Requested vote from peers f649366115124f25bc74e77e773613ad (127.0.98.2:40387), 9203d42b2ff045c386ade97cea8d1430 (127.0.98.3:36421)
I20260925 16:33:29.134496 8229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bce3c53b5474b77b60334022b4a28fa" candidate_uuid: "948c869ed1fe452e8f64517c105e455d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f649366115124f25bc74e77e773613ad"
I20260925 16:33:29.134510 8368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bce3c53b5474b77b60334022b4a28fa" candidate_uuid: "948c869ed1fe452e8f64517c105e455d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9203d42b2ff045c386ade97cea8d1430"
I20260925 16:33:29.134644 8229 raft_consensus.cc:3037] T 4bce3c53b5474b77b60334022b4a28fa P f649366115124f25bc74e77e773613ad [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:33:29.134652 8368 raft_consensus.cc:3037] T 4bce3c53b5474b77b60334022b4a28fa P 9203d42b2ff045c386ade97cea8d1430 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:33:29.135533 8368 raft_consensus.cc:2454] T 4bce3c53b5474b77b60334022b4a28fa P 9203d42b2ff045c386ade97cea8d1430 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948c869ed1fe452e8f64517c105e455d in term 1.
I20260925 16:33:29.135540 8229 raft_consensus.cc:2454] T 4bce3c53b5474b77b60334022b4a28fa P f649366115124f25bc74e77e773613ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948c869ed1fe452e8f64517c105e455d in term 1.
I20260925 16:33:29.135819 8022 leader_election.cc:302] T 4bce3c53b5474b77b60334022b4a28fa P 948c869ed1fe452e8f64517c105e455d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9203d42b2ff045c386ade97cea8d1430, 948c869ed1fe452e8f64517c105e455d; no voters:
I20260925 16:33:29.136013 8454 raft_consensus.cc:2781] T 4bce3c53b5474b77b60334022b4a28fa P 948c869ed1fe452e8f64517c105e455d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:33:29.136143 8454 raft_consensus.cc:686] T 4bce3c53b5474b77b60334022b4a28fa P 948c869ed1fe452e8f64517c105e455d [term 1 LEADER]: Becoming Leader. State: Replica: 948c869ed1fe452e8f64517c105e455d, State: Running, Role: LEADER
I20260925 16:33:29.136302 8454 consensus_queue.cc:237] T 4bce3c53b5474b77b60334022b4a28fa P 948c869ed1fe452e8f64517c105e455d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948c869ed1fe452e8f64517c105e455d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41731 } } peers { permanent_uuid: "f649366115124f25bc74e77e773613ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } } peers { permanent_uuid: "9203d42b2ff045c386ade97cea8d1430" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36421 } }
I20260925 16:33:29.137812 7932 catalog_manager.cc:5495] T 4bce3c53b5474b77b60334022b4a28fa P 948c869ed1fe452e8f64517c105e455d reported cstate change: term changed from 0 to 1, leader changed from <none> to 948c869ed1fe452e8f64517c105e455d (127.0.98.1). New cstate: current_term: 1 leader_uuid: "948c869ed1fe452e8f64517c105e455d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948c869ed1fe452e8f64517c105e455d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f649366115124f25bc74e77e773613ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9203d42b2ff045c386ade97cea8d1430" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36421 } health_report { overall_health: UNKNOWN } } }
I20260925 16:33:29.175623 8483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:29.171990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51908 (local address 127.0.98.1:41731)
0925 16:33:29.172130 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:29.172134 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:29.172147 (+ 13us) server_negotiation.cc:407] Connection header received
0925 16:33:29.172217 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:29.172218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:29.172258 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:29.172321 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:33:29.172601 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:29.173146 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:29.174174 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:29.174645 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:29.174769 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:29.175141 (+ 372us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:33:29.175225 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:33:29.175290 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:29.175449 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 16:33:29.175514 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 16:33:29.181666 8229 raft_consensus.cc:1261] T 4bce3c53b5474b77b60334022b4a28fa P f649366115124f25bc74e77e773613ad [term 1 FOLLOWER]: Refusing update from remote peer 948c869ed1fe452e8f64517c105e455d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:33:29.181665 8368 raft_consensus.cc:1261] T 4bce3c53b5474b77b60334022b4a28fa P 9203d42b2ff045c386ade97cea8d1430 [term 1 FOLLOWER]: Refusing update from remote peer 948c869ed1fe452e8f64517c105e455d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:33:29.182058 8454 consensus_queue.cc:1035] T 4bce3c53b5474b77b60334022b4a28fa P 948c869ed1fe452e8f64517c105e455d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9203d42b2ff045c386ade97cea8d1430" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:33:29.182217 8450 consensus_queue.cc:1035] T 4bce3c53b5474b77b60334022b4a28fa P 948c869ed1fe452e8f64517c105e455d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f649366115124f25bc74e77e773613ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:33:29.214145 7932 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36636:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:33:29.214298 7932 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36636:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:33:29.216313 7932 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7bf7b1c2fbb04b7a96e4038edbe73e92: Sending DeleteTablet for 3 replicas of tablet 4bce3c53b5474b77b60334022b4a28fa
I20260925 16:33:29.216888 8069 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bce3c53b5474b77b60334022b4a28fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:33:29 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51892
I20260925 16:33:29.217010 8209 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bce3c53b5474b77b60334022b4a28fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:33:29 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58520
I20260925 16:33:29.217232 8510 tablet_replica.cc:330] T 4bce3c53b5474b77b60334022b4a28fa P f649366115124f25bc74e77e773613ad: stopping tablet replica
I20260925 16:33:29.217255 8511 tablet_replica.cc:330] T 4bce3c53b5474b77b60334022b4a28fa P 948c869ed1fe452e8f64517c105e455d: stopping tablet replica
I20260925 16:33:29.217365 8510 raft_consensus.cc:2229] T 4bce3c53b5474b77b60334022b4a28fa P f649366115124f25bc74e77e773613ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:33:29.217377 8511 raft_consensus.cc:2229] T 4bce3c53b5474b77b60334022b4a28fa P 948c869ed1fe452e8f64517c105e455d [term 1 LEADER]: Raft consensus shutting down.
I20260925 16:33:29.217399 8349 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bce3c53b5474b77b60334022b4a28fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:33:29 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58224
I20260925 16:33:29.217494 8510 raft_consensus.cc:2258] T 4bce3c53b5474b77b60334022b4a28fa P f649366115124f25bc74e77e773613ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:33:29.217566 8511 raft_consensus.cc:2258] T 4bce3c53b5474b77b60334022b4a28fa P 948c869ed1fe452e8f64517c105e455d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:33:29.217604 8513 tablet_replica.cc:330] T 4bce3c53b5474b77b60334022b4a28fa P 9203d42b2ff045c386ade97cea8d1430: stopping tablet replica
I20260925 16:33:29.217738 8513 raft_consensus.cc:2229] T 4bce3c53b5474b77b60334022b4a28fa P 9203d42b2ff045c386ade97cea8d1430 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:33:29.217875 8513 raft_consensus.cc:2258] T 4bce3c53b5474b77b60334022b4a28fa P 9203d42b2ff045c386ade97cea8d1430 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:33:29.218248 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 8004
I20260925 16:33:29.218297 8511 ts_tablet_manager.cc:1905] T 4bce3c53b5474b77b60334022b4a28fa P 948c869ed1fe452e8f64517c105e455d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:33:29.218403 8510 ts_tablet_manager.cc:1905] T 4bce3c53b5474b77b60334022b4a28fa P f649366115124f25bc74e77e773613ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:33:29.219844 8513 ts_tablet_manager.cc:1905] T 4bce3c53b5474b77b60334022b4a28fa P 9203d42b2ff045c386ade97cea8d1430: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:33:29.220970 8510 ts_tablet_manager.cc:1918] T 4bce3c53b5474b77b60334022b4a28fa P f649366115124f25bc74e77e773613ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 16:33:29.221062 8510 log.cc:1196] T 4bce3c53b5474b77b60334022b4a28fa P f649366115124f25bc74e77e773613ad: Deleting WAL directory at /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/ts-1/wal/wals/4bce3c53b5474b77b60334022b4a28fa
I20260925 16:33:29.221328 8510 ts_tablet_manager.cc:1939] T 4bce3c53b5474b77b60334022b4a28fa P f649366115124f25bc74e77e773613ad: Deleting consensus metadata
I20260925 16:33:29.222002 7916 catalog_manager.cc:4841] TS f649366115124f25bc74e77e773613ad (127.0.98.2:40387): tablet 4bce3c53b5474b77b60334022b4a28fa (table test-table [id=6877bcd387564cfcad0368c7ddbcfb59]) successfully deleted
I20260925 16:33:29.222340 8513 ts_tablet_manager.cc:1918] T 4bce3c53b5474b77b60334022b4a28fa P 9203d42b2ff045c386ade97cea8d1430: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 16:33:29.222429 8513 log.cc:1196] T 4bce3c53b5474b77b60334022b4a28fa P 9203d42b2ff045c386ade97cea8d1430: Deleting WAL directory at /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353966814071-392-0/minicluster-data/ts-2/wal/wals/4bce3c53b5474b77b60334022b4a28fa
I20260925 16:33:29.222714 8513 ts_tablet_manager.cc:1939] T 4bce3c53b5474b77b60334022b4a28fa P 9203d42b2ff045c386ade97cea8d1430: Deleting consensus metadata
I20260925 16:33:29.223377 7918 catalog_manager.cc:4841] TS 9203d42b2ff045c386ade97cea8d1430 (127.0.98.3:36421): tablet 4bce3c53b5474b77b60334022b4a28fa (table test-table [id=6877bcd387564cfcad0368c7ddbcfb59]) successfully deleted
W20260925 16:33:29.228417 7917 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41731 (error 108)
I20260925 16:33:29.228533 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 8144
I20260925 16:33:29.230198 8544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:29.229691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41731 (local address 127.0.0.1:51924)
0925 16:33:29.229993 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:29.230074 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":60,"threads_started":1}
W20260925 16:33:29.230321 7917 catalog_manager.cc:4584] TS 948c869ed1fe452e8f64517c105e455d (127.0.98.1:41731): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4bce3c53b5474b77b60334022b4a28fa: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41731: connect: Connection refused (error 111)
I20260925 16:33:29.235906 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 8284
I20260925 16:33:29.242815 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 7900
2026-09-25T16:33:29Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3311 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790353966814071-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:33:29 dist-test-slave-65pf krb5kdc[8547](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:33:29 dist-test-slave-65pf krb5kdc[8547](info): set up 2 sockets
Sep 25 16:33:29 dist-test-slave-65pf krb5kdc[8547](info): commencing operation
krb5kdc: starting...
W20260925 16:33:30.279316 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790353966814071-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790353966814071-392-0/krb5kdc/test-user.keytab.
Sep 25 16:33:30 dist-test-slave-65pf krb5kdc[8547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:33:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:33:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 16:33:30.400897 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790353966814071-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790353966814071-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790353966814071-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33707
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33707
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:33:30.508822 8560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:33:30.509083 8560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:33:30.509123 8560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:33:30.513088 8560 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:33:30.513144 8560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:33:30.513164 8560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:33:30.513182 8560 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:33:30.513197 8560 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:33:30.518348 8560 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790353966814071-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790353966814071-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33707
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790353966814071-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:33:30.519425 8560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:33:30.520383 8560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:33:30.528628 8568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:30.528635 8566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:30.528767 8560 server_base.cc:1034] running on GCE node
W20260925 16:33:30.528685 8565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:30.529453 8560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:33:30.530547 8560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:33:30.531755 8560 hybrid_clock.cc:648] HybridClock initialized: now 1790354010531727 us; error 39 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-25T16:33:30Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1305 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:33:30 dist-test-slave-65pf krb5kdc[8573](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:33:30 dist-test-slave-65pf krb5kdc[8573](info): set up 2 sockets
Sep 25 16:33:30 dist-test-slave-65pf krb5kdc[8573](info): commencing operation
krb5kdc: starting...
W20260925 16:33:31.599354 392 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/krb5kdc/test-user.keytab.
Sep 25 16:33:31 dist-test-slave-65pf krb5kdc[8573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:33:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:33:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 16:33:31.718042 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41281
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41281
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:33:31.825856 8586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:33:31.826205 8586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:33:31.826285 8586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:33:31.830338 8586 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:33:31.830451 8586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:33:31.830490 8586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:33:31.830526 8586 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:33:31.830560 8586 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:33:31.835850 8586 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8586
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:33:31.836925 8586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:33:31.837961 8586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 16:33:31.846299 8586 server_base.cc:1034] running on GCE node
W20260925 16:33:31.846272 8594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:31.846354 8592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:31.846288 8591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:31.847121 8586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:33:31.848109 8586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:33:31.849295 8586 hybrid_clock.cc:648] HybridClock initialized: now 1790354011849272 us; error 31 us; skew 500 ppm
Sep 25 16:33:31 dist-test-slave-65pf krb5kdc[8573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354011, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:33:31.852029 8586 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 16:33:31.852907 8586 webserver.cc:460] Webserver started at http://127.0.98.62:35089/ using document root <none> and password file <none>
I20260925 16:33:31.853467 8586 fs_manager.cc:359] Metadata directory not provided
I20260925 16:33:31.853528 8586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:33:31.853754 8586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:33:31.855825 8586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/master-0/data/instance:
uuid: "4a4acfb3bc434b22b20cf946fce6e130"
format_stamp: "Formatted at 2026-09-25 16:33:31 on dist-test-slave-65pf"
server_key: "d78e0ea3e852e77b4714ef4334539eb0"
server_key_iv: "41f2ecabf148e37c09d0d747f2c6437f"
server_key_version: "encryptionkey@0"
I20260925 16:33:31.856402 8586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/master-0/wal/instance:
uuid: "4a4acfb3bc434b22b20cf946fce6e130"
format_stamp: "Formatted at 2026-09-25 16:33:31 on dist-test-slave-65pf"
server_key: "d78e0ea3e852e77b4714ef4334539eb0"
server_key_iv: "41f2ecabf148e37c09d0d747f2c6437f"
server_key_version: "encryptionkey@0"
I20260925 16:33:31.860088 8586 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 16:33:31.862386 8601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:31.863706 8586 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:33:31.863844 8586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/master-0/wal
uuid: "4a4acfb3bc434b22b20cf946fce6e130"
format_stamp: "Formatted at 2026-09-25 16:33:31 on dist-test-slave-65pf"
server_key: "d78e0ea3e852e77b4714ef4334539eb0"
server_key_iv: "41f2ecabf148e37c09d0d747f2c6437f"
server_key_version: "encryptionkey@0"
I20260925 16:33:31.863955 8586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:33:31.876750 8586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:33:31.879786 8586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:33:31.879988 8586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:33:31.886698 8586 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41281
I20260925 16:33:31.886732 8653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41281 every 8 connection(s)
I20260925 16:33:31.887673 8586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:33:31.890427 8654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:33:31.893679 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 8586
I20260925 16:33:31.893821 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/master-0/wal/instance
I20260925 16:33:31.894131 392 external_mini_cluster.cc:1428] Setting key fda42489c278cd516d3ec5691e79b49a
I20260925 16:33:31.896513 8654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a4acfb3bc434b22b20cf946fce6e130: Bootstrap starting.
I20260925 16:33:31.899091 8654 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a4acfb3bc434b22b20cf946fce6e130: Neither blocks nor log segments found. Creating new log.
I20260925 16:33:31.899955 8654 log.cc:824] T 00000000000000000000000000000000 P 4a4acfb3bc434b22b20cf946fce6e130: Log is configured to *not* fsync() on all Append() calls
I20260925 16:33:31.901865 8654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a4acfb3bc434b22b20cf946fce6e130: No bootstrap required, opened a new log
Sep 25 16:33:31 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:33:31.905289 8654 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a4acfb3bc434b22b20cf946fce6e130 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a4acfb3bc434b22b20cf946fce6e130" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41281 } }
I20260925 16:33:31.905534 8654 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a4acfb3bc434b22b20cf946fce6e130 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:33:31.905614 8654 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a4acfb3bc434b22b20cf946fce6e130 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a4acfb3bc434b22b20cf946fce6e130, State: Initialized, Role: FOLLOWER
I20260925 16:33:31.906153 8654 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a4acfb3bc434b22b20cf946fce6e130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a4acfb3bc434b22b20cf946fce6e130" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41281 } }
I20260925 16:33:31.906324 8654 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a4acfb3bc434b22b20cf946fce6e130 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:33:31.906389 8654 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a4acfb3bc434b22b20cf946fce6e130 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:33:31.906481 8654 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a4acfb3bc434b22b20cf946fce6e130 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:33:31.907395 8654 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a4acfb3bc434b22b20cf946fce6e130 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a4acfb3bc434b22b20cf946fce6e130" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41281 } }
I20260925 16:33:31.907791 8654 leader_election.cc:302] T 00000000000000000000000000000000 P 4a4acfb3bc434b22b20cf946fce6e130 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a4acfb3bc434b22b20cf946fce6e130; no voters:
I20260925 16:33:31.908186 8654 leader_election.cc:288] T 00000000000000000000000000000000 P 4a4acfb3bc434b22b20cf946fce6e130 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:33:31.908254 8659 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a4acfb3bc434b22b20cf946fce6e130 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:33:31.908437 8659 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a4acfb3bc434b22b20cf946fce6e130 [term 1 LEADER]: Becoming Leader. State: Replica: 4a4acfb3bc434b22b20cf946fce6e130, State: Running, Role: LEADER
I20260925 16:33:31.908779 8659 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a4acfb3bc434b22b20cf946fce6e130 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a4acfb3bc434b22b20cf946fce6e130" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41281 } }
I20260925 16:33:31.908861 8657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:31.895594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60954 (local address 127.0.98.62:41281)
0925 16:33:31.895999 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:31.896030 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:31.896066 (+ 36us) server_negotiation.cc:407] Connection header received
0925 16:33:31.896725 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:31.896743 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:31.897079 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:31.897357 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:31.898259 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:31.899442 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:31.900597 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:31.901076 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:31.901266 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:31.903832 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:31.903837 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:31.903850 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:31.903875 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:31.905652 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:31.906144 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:31.906148 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:31.906153 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:31.906267 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:31.906636 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:31.906641 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:31.906642 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:31.907057 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:31.907282 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:31.907573 (+ 291us) server_negotiation.cc:299] Negotiation successful
0925 16:33:31.907869 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":142,"threads_started":1}
I20260925 16:33:31.909430 8654 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a4acfb3bc434b22b20cf946fce6e130 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:33:31.910511 8661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a4acfb3bc434b22b20cf946fce6e130 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a4acfb3bc434b22b20cf946fce6e130. Latest consensus state: current_term: 1 leader_uuid: "4a4acfb3bc434b22b20cf946fce6e130" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a4acfb3bc434b22b20cf946fce6e130" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41281 } } }
I20260925 16:33:31.910662 8661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a4acfb3bc434b22b20cf946fce6e130 [sys.catalog]: This master's current role is: LEADER
I20260925 16:33:31.911041 8660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a4acfb3bc434b22b20cf946fce6e130 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a4acfb3bc434b22b20cf946fce6e130" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a4acfb3bc434b22b20cf946fce6e130" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41281 } } }
I20260925 16:33:31.911737 8668 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:33:31.911931 8660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a4acfb3bc434b22b20cf946fce6e130 [sys.catalog]: This master's current role is: LEADER
I20260925 16:33:31.914438 8668 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:33:31.919728 8668 catalog_manager.cc:1347] Generated new cluster ID: 2008b42c5b6a4ca3b3088cc6931f098d
I20260925 16:33:31.919811 8668 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:33:31.936630 8668 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:33:31.937559 8668 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:33:31.942420 8668 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a4acfb3bc434b22b20cf946fce6e130: Generated new TSK 0
I20260925 16:33:31.943130 8668 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 16:33:31.967803 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41281
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:33:32.078532 8680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:33:32.078814 8680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:33:32.078855 8680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:33:32.082890 8680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:33:32.082945 8680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:33:32.083004 8680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 16:33:32.088141 8680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:33:32.089123 8680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:33:32.090086 8680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:33:32.100133 8686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:32.100143 8688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:32.100234 8685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:32.100453 8680 server_base.cc:1034] running on GCE node
I20260925 16:33:32.100924 8680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:33:32.101577 8680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:33:32.102759 8680 hybrid_clock.cc:648] HybridClock initialized: now 1790354012102748 us; error 34 us; skew 500 ppm
Sep 25 16:33:32 dist-test-slave-65pf krb5kdc[8573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354012, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:33:32.105458 8680 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 16:33:32.106343 8680 webserver.cc:460] Webserver started at http://127.0.98.1:38157/ using document root <none> and password file <none>
I20260925 16:33:32.106986 8680 fs_manager.cc:359] Metadata directory not provided
I20260925 16:33:32.107079 8680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:33:32.107342 8680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:33:32.109423 8680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-0/data/instance:
uuid: "0a4399702f7f483b9717d351f967131a"
format_stamp: "Formatted at 2026-09-25 16:33:32 on dist-test-slave-65pf"
server_key: "10abf5bafe07305d369eac3d7518993c"
server_key_iv: "4dd2b1fd328c682d3dc39356cbe14aba"
server_key_version: "encryptionkey@0"
I20260925 16:33:32.110081 8680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0a4399702f7f483b9717d351f967131a"
format_stamp: "Formatted at 2026-09-25 16:33:32 on dist-test-slave-65pf"
server_key: "10abf5bafe07305d369eac3d7518993c"
server_key_iv: "4dd2b1fd328c682d3dc39356cbe14aba"
server_key_version: "encryptionkey@0"
I20260925 16:33:32.114197 8680 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260925 16:33:32.116667 8695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:32.117842 8680 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:33:32.117986 8680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-0/wal
uuid: "0a4399702f7f483b9717d351f967131a"
format_stamp: "Formatted at 2026-09-25 16:33:32 on dist-test-slave-65pf"
server_key: "10abf5bafe07305d369eac3d7518993c"
server_key_iv: "4dd2b1fd328c682d3dc39356cbe14aba"
server_key_version: "encryptionkey@0"
I20260925 16:33:32.118101 8680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:33:32.135449 8680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:33:32.138268 8680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:33:32.138484 8680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:33:32.139107 8680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:33:32.139940 8680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:33:32.140014 8680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:32.140085 8680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:33:32.140141 8680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:32.148900 8680 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44025
I20260925 16:33:32.148919 8808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44025 every 8 connection(s)
I20260925 16:33:32.149986 8680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
I20260925 16:33:32.153991 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 8680
I20260925 16:33:32.154150 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-0/wal/instance
I20260925 16:33:32.154500 392 external_mini_cluster.cc:1428] Setting key 3a81df90d42d1a771cb486175f32b316
Sep 25 16:33:32 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354012, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 16:33:32.165019 8657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:32.151851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40195 (local address 127.0.98.62:41281)
0925 16:33:32.151998 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:32.152002 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:32.152886 (+ 884us) server_negotiation.cc:407] Connection header received
0925 16:33:32.154279 (+ 1393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:32.154282 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:32.154346 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:32.154449 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:32.155924 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:32.156693 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:32.158119 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:32.158344 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:32.158426 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:32.161187 (+ 2761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:32.161190 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:32.161192 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:32.161215 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:32.162796 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:32.163414 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:32.163417 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:32.163418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:32.163491 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:32.163884 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:32.163886 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:32.163887 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:32.164054 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:32.164127 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:32.164782 (+ 655us) server_negotiation.cc:299] Negotiation successful
0925 16:33:32.164892 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 16:33:32.165906 8811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:32.152297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41281 (local address 127.0.98.1:40195)
0925 16:33:32.152727 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:32.152770 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:32.154005 (+ 1235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:32.154653 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:32.154665 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:32.155133 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:32.155775 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:32.155794 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:32.156821 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:32.156824 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:32.157990 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:32.158003 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:32.158507 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:32.158512 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:32.158756 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:32.159424 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:32.159439 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:32.160990 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:33:32.162910 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:32.162917 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:32.162936 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:32.163311 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:32.163584 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:32.163587 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:32.163591 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:32.163784 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:32.164164 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:33:32.164170 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:33:32.164505 (+ 335us) client_negotiation.cc:769] Sending connection context
0925 16:33:32.164734 (+ 229us) client_negotiation.cc:240] Negotiation successful
0925 16:33:32.164998 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":113,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260925 16:33:32.167367 8809 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41281
I20260925 16:33:32.167614 8809 heartbeater.cc:464] Registering TS with master...
I20260925 16:33:32.168162 8809 heartbeater.cc:511] Master 127.0.98.62:41281 requested a full tablet report, sending...
I20260925 16:33:32.169778 8618 ts_manager.cc:194] Registered new tserver with Master: 0a4399702f7f483b9717d351f967131a (127.0.98.1:44025)
I20260925 16:33:32.171062 8618 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40195
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 16:33:32.181653 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41281
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:33:32.290869 8814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:33:32.291189 8814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:33:32.291280 8814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:33:32.295414 8814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:33:32.295501 8814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:33:32.295595 8814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 16:33:32.300849 8814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:33:32.301950 8814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:33:32.303025 8814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:33:32.312958 8822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:32.312942 8819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:32.313050 8814 server_base.cc:1034] running on GCE node
W20260925 16:33:32.312942 8820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:32.313627 8814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:33:32.314293 8814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:33:32.315474 8814 hybrid_clock.cc:648] HybridClock initialized: now 1790354012315430 us; error 56 us; skew 500 ppm
Sep 25 16:33:32 dist-test-slave-65pf krb5kdc[8573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354012, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:33:32.318226 8814 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 16:33:32.319200 8814 webserver.cc:460] Webserver started at http://127.0.98.2:35377/ using document root <none> and password file <none>
I20260925 16:33:32.319768 8814 fs_manager.cc:359] Metadata directory not provided
I20260925 16:33:32.319859 8814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:33:32.320089 8814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:33:32.322242 8814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-1/data/instance:
uuid: "9ca0581a83f740b88710f5f6652ccb32"
format_stamp: "Formatted at 2026-09-25 16:33:32 on dist-test-slave-65pf"
server_key: "20088b30eb4250fbc3115e7554d37a64"
server_key_iv: "917b95cec3a990cd35f56976959e32c0"
server_key_version: "encryptionkey@0"
I20260925 16:33:32.322847 8814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9ca0581a83f740b88710f5f6652ccb32"
format_stamp: "Formatted at 2026-09-25 16:33:32 on dist-test-slave-65pf"
server_key: "20088b30eb4250fbc3115e7554d37a64"
server_key_iv: "917b95cec3a990cd35f56976959e32c0"
server_key_version: "encryptionkey@0"
I20260925 16:33:32.326536 8814 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 16:33:32.329068 8829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:32.330303 8814 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:33:32.330448 8814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-1/wal
uuid: "9ca0581a83f740b88710f5f6652ccb32"
format_stamp: "Formatted at 2026-09-25 16:33:32 on dist-test-slave-65pf"
server_key: "20088b30eb4250fbc3115e7554d37a64"
server_key_iv: "917b95cec3a990cd35f56976959e32c0"
server_key_version: "encryptionkey@0"
I20260925 16:33:32.330564 8814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:33:32.344741 8814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:33:32.348136 8814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:33:32.348345 8814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:33:32.348932 8814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:33:32.349771 8814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:33:32.349843 8814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:32.349913 8814 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:33:32.349965 8814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:32.360646 8814 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39877
I20260925 16:33:32.361789 8814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
I20260925 16:33:32.361800 8942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39877 every 8 connection(s)
I20260925 16:33:32.368129 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 8814
I20260925 16:33:32.368281 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-1/wal/instance
I20260925 16:33:32.368672 392 external_mini_cluster.cc:1428] Setting key 0a22a11ac1687ad1e93b745f7ef9504e
Sep 25 16:33:32 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354012, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:33:32.376528 8657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:32.365299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47519 (local address 127.0.98.62:41281)
0925 16:33:32.365430 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:32.365435 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:32.365460 (+ 25us) server_negotiation.cc:407] Connection header received
0925 16:33:32.366402 (+ 942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:32.366404 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:32.366446 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:32.366520 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:32.367807 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:32.368414 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:32.369886 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:32.370161 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:32.370273 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:32.372855 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:32.372862 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:32.372866 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:32.372895 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:32.374345 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:32.375001 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:32.375004 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:32.375006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:32.375089 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:32.375440 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:32.375443 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:32.375445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:32.375633 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:32.375742 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:32.376238 (+ 496us) server_negotiation.cc:299] Negotiation successful
0925 16:33:32.376363 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 16:33:32.377362 8945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:32.364486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41281 (local address 127.0.98.2:47519)
0925 16:33:32.364947 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:32.364989 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:32.366221 (+ 1232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:32.366720 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:32.366726 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:32.367086 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:32.367672 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:32.367692 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:32.368544 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:32.368548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:32.369739 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:32.369749 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:32.370310 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:32.370314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:32.370485 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:32.371142 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:32.371158 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:32.372661 (+ 1503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:33:32.374460 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:32.374466 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:32.374482 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:32.374891 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:32.375190 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:32.375194 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:32.375196 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:32.375339 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:32.375737 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:33:32.375744 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:33:32.376001 (+ 257us) client_negotiation.cc:769] Sending connection context
0925 16:33:32.376189 (+ 188us) client_negotiation.cc:240] Negotiation successful
0925 16:33:32.376512 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":119,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260925 16:33:32.378536 8943 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41281
I20260925 16:33:32.378839 8943 heartbeater.cc:464] Registering TS with master...
I20260925 16:33:32.379364 8943 heartbeater.cc:511] Master 127.0.98.62:41281 requested a full tablet report, sending...
I20260925 16:33:32.380451 8618 ts_manager.cc:194] Registered new tserver with Master: 9ca0581a83f740b88710f5f6652ccb32 (127.0.98.2:39877)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260925 16:33:32.381207 8618 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47519
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 16:33:32.395358 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41281
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:33:32.505846 8948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:33:32.506114 8948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:33:32.506197 8948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:33:32.510509 8948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:33:32.510572 8948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:33:32.510738 8948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 16:33:32.516170 8948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:33:32.517283 8948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:33:32.518342 8948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:33:32.527964 8953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:32.528236 8956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:32.527930 8954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:32.528390 8948 server_base.cc:1034] running on GCE node
I20260925 16:33:32.528787 8948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:33:32.529448 8948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:33:32.530652 8948 hybrid_clock.cc:648] HybridClock initialized: now 1790354012530630 us; error 34 us; skew 500 ppm
Sep 25 16:33:32 dist-test-slave-65pf krb5kdc[8573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354012, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:33:32.533396 8948 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 16:33:32.534264 8948 webserver.cc:460] Webserver started at http://127.0.98.3:34997/ using document root <none> and password file <none>
I20260925 16:33:32.534862 8948 fs_manager.cc:359] Metadata directory not provided
I20260925 16:33:32.534925 8948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:33:32.535091 8948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:33:32.537050 8948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-2/data/instance:
uuid: "f25b083d77434787be8c0e8d07657cf5"
format_stamp: "Formatted at 2026-09-25 16:33:32 on dist-test-slave-65pf"
server_key: "e6af055db52c8550dcf36b8f3ed512c6"
server_key_iv: "18cf83328fda6606c901a533a256f04c"
server_key_version: "encryptionkey@0"
I20260925 16:33:32.537571 8948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f25b083d77434787be8c0e8d07657cf5"
format_stamp: "Formatted at 2026-09-25 16:33:32 on dist-test-slave-65pf"
server_key: "e6af055db52c8550dcf36b8f3ed512c6"
server_key_iv: "18cf83328fda6606c901a533a256f04c"
server_key_version: "encryptionkey@0"
I20260925 16:33:32.541237 8948 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 16:33:32.543658 8963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:32.544852 8948 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:33:32.544947 8948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-2/wal
uuid: "f25b083d77434787be8c0e8d07657cf5"
format_stamp: "Formatted at 2026-09-25 16:33:32 on dist-test-slave-65pf"
server_key: "e6af055db52c8550dcf36b8f3ed512c6"
server_key_iv: "18cf83328fda6606c901a533a256f04c"
server_key_version: "encryptionkey@0"
I20260925 16:33:32.545029 8948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:33:32.559904 8948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:33:32.562803 8948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:33:32.562948 8948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:33:32.563473 8948 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:33:32.564221 8948 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:33:32.564267 8948 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:32.564306 8948 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:33:32.564327 8948 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:32.573179 8948 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40921
I20260925 16:33:32.573233 9076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40921 every 8 connection(s)
I20260925 16:33:32.574097 8948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
I20260925 16:33:32.581295 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 8948
I20260925 16:33:32.581391 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-2/wal/instance
I20260925 16:33:32.581677 392 external_mini_cluster.cc:1428] Setting key cc852f779f06af7af6d941a514ff38ec
Sep 25 16:33:32 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354012, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:33:32.587409 8657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:32.575916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35319 (local address 127.0.98.62:41281)
0925 16:33:32.576090 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:32.576095 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:32.576899 (+ 804us) server_negotiation.cc:407] Connection header received
0925 16:33:32.578243 (+ 1344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:32.578246 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:32.578294 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:32.578375 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:32.579559 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:32.580135 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:32.581230 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:32.581562 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:32.581672 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:32.584288 (+ 2616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:32.584291 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:32.584293 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:32.584317 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:32.585486 (+ 1169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:32.585931 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:32.585934 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:32.585935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:32.586008 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:32.586352 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:32.586354 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:32.586356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:32.586545 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:32.586736 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:32.587183 (+ 447us) server_negotiation.cc:299] Negotiation successful
0925 16:33:32.587294 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 16:33:32.588240 9079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:32.576314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41281 (local address 127.0.98.3:35319)
0925 16:33:32.576747 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:32.576790 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:32.578025 (+ 1235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:32.578534 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:32.578541 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:32.578950 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:32.579421 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:32.579434 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:32.580243 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:32.580245 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:32.581091 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:32.581099 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:32.581716 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:32.581720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:32.581889 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:32.582500 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:32.582515 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:32.584129 (+ 1614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:33:32.585594 (+ 1465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:32.585598 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:32.585608 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:32.585848 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:32.586090 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:32.586092 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:32.586093 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:32.586223 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:32.586711 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:33:32.586726 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:33:32.586946 (+ 220us) client_negotiation.cc:769] Sending connection context
0925 16:33:32.587102 (+ 156us) client_negotiation.cc:240] Negotiation successful
0925 16:33:32.587358 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":118,"threads_started":1}
I20260925 16:33:32.589444 9077 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41281
I20260925 16:33:32.589684 9077 heartbeater.cc:464] Registering TS with master...
I20260925 16:33:32.590153 9077 heartbeater.cc:511] Master 127.0.98.62:41281 requested a full tablet report, sending...
I20260925 16:33:32.591154 8618 ts_manager.cc:194] Registered new tserver with Master: f25b083d77434787be8c0e8d07657cf5 (127.0.98.3:40921)
I20260925 16:33:32.591712 8618 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35319
I20260925 16:33:32.595570 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260925 16:33:32.665246 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 8680
I20260925 16:33:32.671087 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 8814
I20260925 16:33:32.676223 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 8948
I20260925 16:33:32.681380 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 8586
I20260925 16:33:32.687506 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41281
--webserver_interface=127.0.98.62
--webserver_port=35089
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41281
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:33:32.796775 9086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:33:32.797101 9086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:33:32.797183 9086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:33:32.801249 9086 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:33:32.801357 9086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:33:32.801395 9086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:33:32.801434 9086 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:33:32.801466 9086 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:33:32.806782 9086 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=35089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:33:32.807844 9086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:33:32.808851 9086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:33:32.816964 9091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:32.816975 9092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:32.816982 9094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:32.817260 9086 server_base.cc:1034] running on GCE node
I20260925 16:33:32.817823 9086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:33:32.818930 9086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:33:32.820107 9086 hybrid_clock.cc:648] HybridClock initialized: now 1790354012820072 us; error 46 us; skew 500 ppm
Sep 25 16:33:32 dist-test-slave-65pf krb5kdc[8573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354012, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:33:32.822817 9086 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 16:33:32.823612 9086 webserver.cc:460] Webserver started at http://127.0.98.62:35089/ using document root <none> and password file <none>
I20260925 16:33:32.824122 9086 fs_manager.cc:359] Metadata directory not provided
I20260925 16:33:32.824191 9086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:33:32.827826 9086 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:33:32.829797 9101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:32.831099 9086 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:33:32.831235 9086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/master-0/wal
uuid: "4a4acfb3bc434b22b20cf946fce6e130"
format_stamp: "Formatted at 2026-09-25 16:33:31 on dist-test-slave-65pf"
server_key: "d78e0ea3e852e77b4714ef4334539eb0"
server_key_iv: "41f2ecabf148e37c09d0d747f2c6437f"
server_key_version: "encryptionkey@0"
I20260925 16:33:32.831768 9086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:33:32.857906 9086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:33:32.860846 9086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:33:32.861063 9086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:33:32.868053 9086 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41281
I20260925 16:33:32.868077 9153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41281 every 8 connection(s)
I20260925 16:33:32.869138 9086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:33:32.873627 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 9086
I20260925 16:33:32.873986 9154 sys_catalog.cc:263] Verifying existing consensus state
I20260925 16:33:32.874238 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44025
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:41281
--webserver_port=38157
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 16:33:32.876533 9154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a4acfb3bc434b22b20cf946fce6e130: Bootstrap starting.
I20260925 16:33:32.893599 9154 log.cc:824] T 00000000000000000000000000000000 P 4a4acfb3bc434b22b20cf946fce6e130: Log is configured to *not* fsync() on all Append() calls
I20260925 16:33:32.900411 9154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a4acfb3bc434b22b20cf946fce6e130: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 16:33:32.900830 9154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a4acfb3bc434b22b20cf946fce6e130: Bootstrap complete.
I20260925 16:33:32.905100 9154 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a4acfb3bc434b22b20cf946fce6e130 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a4acfb3bc434b22b20cf946fce6e130" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41281 } }
I20260925 16:33:32.905495 9154 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a4acfb3bc434b22b20cf946fce6e130 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a4acfb3bc434b22b20cf946fce6e130, State: Initialized, Role: FOLLOWER
I20260925 16:33:32.906010 9154 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a4acfb3bc434b22b20cf946fce6e130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a4acfb3bc434b22b20cf946fce6e130" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41281 } }
I20260925 16:33:32.906167 9154 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a4acfb3bc434b22b20cf946fce6e130 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:33:32.906240 9154 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a4acfb3bc434b22b20cf946fce6e130 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:33:32.906365 9154 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a4acfb3bc434b22b20cf946fce6e130 [term 1 FOLLOWER]: Advancing to term 2
I20260925 16:33:32.908432 9154 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a4acfb3bc434b22b20cf946fce6e130 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a4acfb3bc434b22b20cf946fce6e130" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41281 } }
I20260925 16:33:32.908850 9154 leader_election.cc:302] T 00000000000000000000000000000000 P 4a4acfb3bc434b22b20cf946fce6e130 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a4acfb3bc434b22b20cf946fce6e130; no voters:
I20260925 16:33:32.909225 9154 leader_election.cc:288] T 00000000000000000000000000000000 P 4a4acfb3bc434b22b20cf946fce6e130 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 16:33:32.909324 9158 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a4acfb3bc434b22b20cf946fce6e130 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 16:33:32.909540 9158 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a4acfb3bc434b22b20cf946fce6e130 [term 2 LEADER]: Becoming Leader. State: Replica: 4a4acfb3bc434b22b20cf946fce6e130, State: Running, Role: LEADER
I20260925 16:33:32.909847 9158 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a4acfb3bc434b22b20cf946fce6e130 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a4acfb3bc434b22b20cf946fce6e130" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41281 } }
I20260925 16:33:32.910498 9154 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a4acfb3bc434b22b20cf946fce6e130 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:33:32.910857 9158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a4acfb3bc434b22b20cf946fce6e130 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a4acfb3bc434b22b20cf946fce6e130. Latest consensus state: current_term: 2 leader_uuid: "4a4acfb3bc434b22b20cf946fce6e130" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a4acfb3bc434b22b20cf946fce6e130" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41281 } } }
I20260925 16:33:32.910985 9158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a4acfb3bc434b22b20cf946fce6e130 [sys.catalog]: This master's current role is: LEADER
I20260925 16:33:32.911345 9159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a4acfb3bc434b22b20cf946fce6e130 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4a4acfb3bc434b22b20cf946fce6e130" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a4acfb3bc434b22b20cf946fce6e130" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41281 } } }
I20260925 16:33:32.911507 9159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a4acfb3bc434b22b20cf946fce6e130 [sys.catalog]: This master's current role is: LEADER
I20260925 16:33:32.911360 9162 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:33:32.915244 9162 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:33:32.916124 9162 catalog_manager.cc:1259] Loaded cluster ID: 2008b42c5b6a4ca3b3088cc6931f098d
I20260925 16:33:32.916189 9162 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:33:32.918710 9162 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:33:32.919454 9162 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4a4acfb3bc434b22b20cf946fce6e130: Loaded TSK: 0
I20260925 16:33:32.920702 9162 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 16:33:32.988735 9156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:33:32.989002 9156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:33:32.989048 9156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:33:32.993132 9156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:33:32.993201 9156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:33:32.993263 9156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 16:33:32.998476 9156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44025
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=38157
--tserver_master_addrs=127.0.98.62:41281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:33:32.999554 9156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:33:33.000535 9156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:33:33.011047 9180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:33.011067 9181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:33.011262 9156 server_base.cc:1034] running on GCE node
W20260925 16:33:33.011047 9183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:33.011919 9156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:33:33.012552 9156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:33:33.013734 9156 hybrid_clock.cc:648] HybridClock initialized: now 1790354013013695 us; error 68 us; skew 500 ppm
Sep 25 16:33:33 dist-test-slave-65pf krb5kdc[8573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354013, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:33:33.016538 9156 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260925 16:33:33.017427 9156 webserver.cc:460] Webserver started at http://127.0.98.1:38157/ using document root <none> and password file <none>
I20260925 16:33:33.017980 9156 fs_manager.cc:359] Metadata directory not provided
I20260925 16:33:33.018051 9156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:33:33.022729 9156 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 16:33:33.024848 9190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:33.026142 9156 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:33:33.026283 9156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-0/wal
uuid: "0a4399702f7f483b9717d351f967131a"
format_stamp: "Formatted at 2026-09-25 16:33:32 on dist-test-slave-65pf"
server_key: "10abf5bafe07305d369eac3d7518993c"
server_key_iv: "4dd2b1fd328c682d3dc39356cbe14aba"
server_key_version: "encryptionkey@0"
I20260925 16:33:33.026893 9156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:33:33.045295 9156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:33:33.048516 9156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:33:33.048722 9156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:33:33.049307 9156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:33:33.050192 9156 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:33:33.050261 9156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:33.050329 9156 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:33:33.050375 9156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:33.061199 9156 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44025
I20260925 16:33:33.061264 9303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44025 every 8 connection(s)
I20260925 16:33:33.062234 9156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
I20260925 16:33:33.071362 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 9156
I20260925 16:33:33.071849 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:39877
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:41281
--webserver_port=35377
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 16:33:33 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:33:33 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:33:33.075208 9307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:33.064371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41281 (local address 127.0.98.1:42493)
0925 16:33:33.064807 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:33.064857 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:33.066123 (+ 1266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:33.067243 (+ 1120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:33.067249 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:33.067596 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:33.068033 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:33.068046 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:33.069254 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:33.069256 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:33.070096 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:33.070104 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:33.070859 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:33.070862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:33.070991 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:33.071649 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:33.071665 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:33.074341 (+ 2676us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41281: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":143,"threads_started":1}
W20260925 16:33:33.075554 9306 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:33:33.064221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42493 (local address 127.0.98.62:41281)
0925 16:33:33.064637 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:33.064652 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:33.064974 (+ 322us) server_negotiation.cc:407] Connection header received
0925 16:33:33.066498 (+ 1524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:33.066509 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:33.066878 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:33.067151 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:33.068198 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:33.069142 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:33.070205 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:33.070720 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:33.070835 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:33.074707 (+ 3872us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42493: BlockingRecv error: recv got EOF from 127.0.98.1:42493 (error 108)
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":126,"threads_started":1}
W20260925 16:33:33.076043 9304 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41281 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:41281: Client connection negotiation failed: client connection to 127.0.98.62:41281: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 25 16:33:33 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:33:33 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:33:33.082432 9307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:33.076483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41281 (local address 127.0.98.1:39077)
0925 16:33:33.076658 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:33.076677 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:33.076769 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:33.077024 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:33.077029 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:33.077219 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:33.077371 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:33.077379 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:33.078087 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:33.078092 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:33.079167 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:33.079182 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:33.079619 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:33.079625 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:33.079749 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:33.080259 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:33.080280 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:33.082336 (+ 2056us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41281: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":68}
W20260925 16:33:33.082737 9306 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:33:33.076598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39077 (local address 127.0.98.62:41281)
0925 16:33:33.076700 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:33.076703 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:33.076718 (+ 15us) server_negotiation.cc:407] Connection header received
0925 16:33:33.076870 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:33.076872 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:33.076909 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:33.076976 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:33.077494 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:33.077978 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:33.079290 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:33.079498 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:33.079576 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:33.082554 (+ 2978us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:39077: BlockingRecv error: recv got EOF from 127.0.98.1:39077 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Sep 25 16:33:33 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:33:33 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:33:33.088335 9307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:33.083176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41281 (local address 127.0.98.1:50117)
0925 16:33:33.083310 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:33.083328 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:33.083390 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:33.083633 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:33.083636 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:33.083757 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:33.083827 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:33.083831 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:33.084826 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:33.084828 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:33.085333 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:33.085339 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:33.085866 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:33.085868 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:33.085925 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:33.086406 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:33.086417 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:33.088209 (+ 1792us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41281: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 16:33:33.088446 9306 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:33:33.083246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50117 (local address 127.0.98.62:41281)
0925 16:33:33.083397 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:33.083404 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:33.083425 (+ 21us) server_negotiation.cc:407] Connection header received
0925 16:33:33.083469 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:33.083472 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:33.083526 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:33.083610 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:33.083929 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:33.084725 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:33.085433 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:33.085772 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:33.085867 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:33.088328 (+ 2461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50117: BlockingRecv error: recv got EOF from 127.0.98.1:50117 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260925 16:33:33.088649 9304 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 16:33:33.185493 9308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:33:33.185771 9308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:33:33.185815 9308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:33:33.190021 9308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:33:33.190090 9308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:33:33.190198 9308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 16:33:33.195538 9308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:39877
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=35377
--tserver_master_addrs=127.0.98.62:41281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:33:33.196619 9308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:33:33.197659 9308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:33:33.207531 9316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:33.207536 9314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:33.207536 9313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:33.208062 9308 server_base.cc:1034] running on GCE node
I20260925 16:33:33.208503 9308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:33:33.209208 9308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:33:33.210392 9308 hybrid_clock.cc:648] HybridClock initialized: now 1790354013210373 us; error 31 us; skew 500 ppm
Sep 25 16:33:33 dist-test-slave-65pf krb5kdc[8573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354013, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:33:33.213244 9308 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260925 16:33:33.214231 9308 webserver.cc:460] Webserver started at http://127.0.98.2:35377/ using document root <none> and password file <none>
I20260925 16:33:33.214871 9308 fs_manager.cc:359] Metadata directory not provided
I20260925 16:33:33.214988 9308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:33:33.219040 9308 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 16:33:33.221094 9323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:33.222378 9308 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:33:33.222513 9308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-1/wal
uuid: "9ca0581a83f740b88710f5f6652ccb32"
format_stamp: "Formatted at 2026-09-25 16:33:32 on dist-test-slave-65pf"
server_key: "20088b30eb4250fbc3115e7554d37a64"
server_key_iv: "917b95cec3a990cd35f56976959e32c0"
server_key_version: "encryptionkey@0"
I20260925 16:33:33.223140 9308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:33:33.241663 9308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:33:33.244757 9308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:33:33.244969 9308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:33:33.245574 9308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:33:33.246405 9308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:33:33.246475 9308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:33.246547 9308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:33:33.246622 9308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:33.255232 9308 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39877
I20260925 16:33:33.255259 9436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39877 every 8 connection(s)
I20260925 16:33:33.256248 9308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
I20260925 16:33:33.258719 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 9308
I20260925 16:33:33.259212 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:40921
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:41281
--webserver_port=34997
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 16:33:33 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:33:33 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 16:33:33.267409 9306 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:33:33.258004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34441 (local address 127.0.98.62:41281)
0925 16:33:33.258142 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:33.258146 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:33.259080 (+ 934us) server_negotiation.cc:407] Connection header received
0925 16:33:33.260532 (+ 1452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:33.260535 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:33.260596 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:33.260703 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:33.261868 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:33.262401 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:33.263588 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:33.263810 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:33.263922 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:33.267283 (+ 3361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34441: BlockingRecv error: recv got EOF from 127.0.98.2:34441 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 16:33:33.268038 9439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:33.258337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41281 (local address 127.0.98.2:34441)
0925 16:33:33.258940 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:33.258983 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:33.260307 (+ 1324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:33.260890 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:33.260898 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:33.261247 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:33.261721 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:33.261734 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:33.262524 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:33.262527 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:33.263461 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:33.263470 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:33.263961 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:33.263965 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:33.264142 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:33.264795 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:33.264811 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:33.267160 (+ 2349us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41281: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":189,"threads_started":1}
W20260925 16:33:33.268515 9437 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41281 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:41281: Client connection negotiation failed: client connection to 127.0.98.62:41281: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 25 16:33:33 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:33:33 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:33:33.274127 9439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:33.268998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41281 (local address 127.0.98.2:42391)
0925 16:33:33.269111 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:33.269138 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:33.269219 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:33.269475 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:33.269477 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:33.269650 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:33.269732 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:33.269737 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:33.270442 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:33.270449 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:33.271075 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:33.271083 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:33.271520 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:33.271523 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:33.271608 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:33.272114 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:33.272125 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:33.274001 (+ 1876us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41281: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 16:33:33.274330 9306 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:33:33.269070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42391 (local address 127.0.98.62:41281)
0925 16:33:33.269187 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:33.269192 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:33.269204 (+ 12us) server_negotiation.cc:407] Connection header received
0925 16:33:33.269328 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:33.269331 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:33.269378 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:33.269461 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:33.269842 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:33.270338 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:33.271196 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:33.271413 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:33.271521 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:33.274156 (+ 2635us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42391: BlockingRecv error: recv got EOF from 127.0.98.2:42391 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 25 16:33:33 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:33:33 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:33:33.280097 9439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:33.274849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41281 (local address 127.0.98.2:51277)
0925 16:33:33.274957 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:33.274971 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:33.275045 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:33.275274 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:33.275276 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:33.275431 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:33.275514 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:33.275520 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:33.276188 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:33.276192 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:33.277080 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:33.277090 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:33.277485 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:33.277489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:33.277570 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:33.278012 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:33.278025 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:33.279977 (+ 1952us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41281: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 16:33:33.280233 9306 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:33:33.274910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51277 (local address 127.0.98.62:41281)
0925 16:33:33.275012 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:33.275015 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:33.275028 (+ 13us) server_negotiation.cc:407] Connection header received
0925 16:33:33.275136 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:33.275138 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:33.275172 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:33.275238 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:33.275616 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:33.276094 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:33.277191 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:33.277386 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:33.277459 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:33.280097 (+ 2638us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:51277: BlockingRecv error: recv got EOF from 127.0.98.2:51277 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260925 16:33:33.280699 9437 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 16:33:33.374459 9440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:33:33.374797 9440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:33:33.374867 9440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:33:33.379036 9440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:33:33.379118 9440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:33:33.379180 9440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 16:33:33.384594 9440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:40921
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=34997
--tserver_master_addrs=127.0.98.62:41281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:33:33.385728 9440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:33:33.386801 9440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:33:33.396922 9448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:33.396890 9445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:33.396916 9446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:33.397109 9440 server_base.cc:1034] running on GCE node
I20260925 16:33:33.397648 9440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:33:33.398335 9440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:33:33.399526 9440 hybrid_clock.cc:648] HybridClock initialized: now 1790354013399505 us; error 33 us; skew 500 ppm
Sep 25 16:33:33 dist-test-slave-65pf krb5kdc[8573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354013, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:33:33.402383 9440 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260925 16:33:33.403509 9440 webserver.cc:460] Webserver started at http://127.0.98.3:34997/ using document root <none> and password file <none>
I20260925 16:33:33.404090 9440 fs_manager.cc:359] Metadata directory not provided
I20260925 16:33:33.404198 9440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:33:33.408224 9440 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260925 16:33:33.410221 9455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:33.411504 9440 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 16:33:33.411652 9440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-2/wal
uuid: "f25b083d77434787be8c0e8d07657cf5"
format_stamp: "Formatted at 2026-09-25 16:33:32 on dist-test-slave-65pf"
server_key: "e6af055db52c8550dcf36b8f3ed512c6"
server_key_iv: "18cf83328fda6606c901a533a256f04c"
server_key_version: "encryptionkey@0"
I20260925 16:33:33.412248 9440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:33:33.440778 9440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:33:33.444152 9440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:33:33.444358 9440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:33:33.444963 9440 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:33:33.445854 9440 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:33:33.445925 9440 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:33.445996 9440 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:33:33.446039 9440 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:33.455044 9440 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40921
I20260925 16:33:33.455075 9568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40921 every 8 connection(s)
I20260925 16:33:33.456104 9440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
Sep 25 16:33:33 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:33:33 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:33:33.465889 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 9440
W20260925 16:33:33.466472 9306 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:33:33.457826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34483 (local address 127.0.98.62:41281)
0925 16:33:33.457962 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:33.457966 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:33.458752 (+ 786us) server_negotiation.cc:407] Connection header received
0925 16:33:33.460021 (+ 1269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:33.460025 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:33.460095 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:33.460172 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:33.461283 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:33.461796 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:33.462886 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:33.463116 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:33.463197 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:33.466350 (+ 3153us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:34483: BlockingRecv error: recv got EOF from 127.0.98.3:34483 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 16:33:33.467198 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:33.458152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41281 (local address 127.0.98.3:34483)
0925 16:33:33.458597 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:33.458639 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:33.459823 (+ 1184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:33.460331 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:33.460338 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:33.460686 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:33.461149 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:33.461161 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:33.461894 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:33.461897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:33.462784 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:33.462793 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:33.463242 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:33.463244 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:33.463377 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:33.463903 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:33.463918 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:33.466242 (+ 2324us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41281: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":114,"threads_started":1}
W20260925 16:33:33.467687 9569 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41281 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:41281: Client connection negotiation failed: client connection to 127.0.98.62:41281: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 25 16:33:33 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:33:33 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:33:33.473366 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:33.468129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41281 (local address 127.0.98.3:51121)
0925 16:33:33.468217 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:33.468232 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:33.468300 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:33.468650 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:33.468653 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:33.468819 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:33.468923 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:33.468929 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:33.469617 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:33.469624 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:33.470417 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:33.470449 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:33.470969 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:33.470974 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:33.471095 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:33.471572 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:33.471583 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:33.473232 (+ 1649us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41281: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260925 16:33:33.473532 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:33:33.468211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51121 (local address 127.0.98.62:41281)
0925 16:33:33.468464 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:33.468467 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:33.468477 (+ 10us) server_negotiation.cc:407] Connection header received
0925 16:33:33.468502 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:33.468504 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:33.468534 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:33.468640 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:33.469031 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:33.469510 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:33.470621 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:33.470852 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:33.470936 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:33.473387 (+ 2451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51121: BlockingRecv error: recv got EOF from 127.0.98.3:51121 (error 108)
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":104,"threads_started":1}
I20260925 16:33:33.475998 9306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:33.466927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60968 (local address 127.0.98.62:41281)
0925 16:33:33.467060 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:33.467064 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:33.467077 (+ 13us) server_negotiation.cc:407] Connection header received
0925 16:33:33.467183 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:33.467186 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:33.467230 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:33.467315 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:33.468160 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:33.468731 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:33.469677 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:33.469869 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:33.469938 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:33.471575 (+ 1637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:33.471580 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:33.471595 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:33.471619 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:33.473707 (+ 2088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:33.474214 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:33.474219 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:33.474224 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:33.474381 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:33.474724 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:33.474726 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:33.474727 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:33.475147 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:33.475333 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:33.475652 (+ 319us) server_negotiation.cc:299] Negotiation successful
0925 16:33:33.475895 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Sep 25 16:33:33 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:33:33 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:33:33.479313 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:33.473901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41281 (local address 127.0.98.3:54029)
0925 16:33:33.474225 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:33.474239 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:33.474311 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:33.474603 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:33.474606 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:33.474776 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:33.474847 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:33.474852 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:33.475795 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:33.475798 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:33.476403 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:33.476412 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:33.476916 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:33.476918 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:33.477004 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:33.477377 (+ 373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:33.477393 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:33.479219 (+ 1826us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41281: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":93,"threads_started":1}
W20260925 16:33:33.479424 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:33:33.473953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54029 (local address 127.0.98.62:41281)
0925 16:33:33.474106 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:33.474115 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:33.474305 (+ 190us) server_negotiation.cc:407] Connection header received
0925 16:33:33.474378 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:33.474381 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:33.474427 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:33.474548 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:33.474947 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:33.475660 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:33.476506 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:33.476806 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:33.476905 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:33.479326 (+ 2421us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54029: BlockingRecv error: recv got EOF from 127.0.98.3:54029 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
W20260925 16:33:33.479544 9569 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 25 16:33:34 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:33:34 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:33:34.094472 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:34.089317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41281 (local address 127.0.98.1:60187)
0925 16:33:34.089609 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:34.089623 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:34.089738 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:34.089973 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:34.089975 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:34.090156 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:34.090279 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:34.090284 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:34.091054 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:34.091056 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:34.091608 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:34.091615 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:34.092003 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:34.092006 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:34.092079 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:34.092575 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:34.092593 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:34.094349 (+ 1756us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41281: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":86,"threads_started":1}
W20260925 16:33:34.094691 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:33:34.089377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60187 (local address 127.0.98.62:41281)
0925 16:33:34.089617 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:34.089620 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:34.089693 (+ 73us) server_negotiation.cc:407] Connection header received
0925 16:33:34.089818 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:34.089820 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:34.089862 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:34.089938 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:34.090377 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:34.090965 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:34.091702 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:34.091913 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:34.091985 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:34.094487 (+ 2502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:60187: BlockingRecv error: recv got EOF from 127.0.98.1:60187 (error 108)
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":72,"threads_started":1}
Sep 25 16:33:34 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:33:34 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:33:34.286556 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:34.281471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41281 (local address 127.0.98.2:57881)
0925 16:33:34.281735 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:34.281748 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:34.281858 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:34.282076 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:34.282078 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:34.282253 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:34.282369 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:34.282374 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:34.283094 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:34.283097 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:34.283655 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:34.283663 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:34.284062 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:34.284065 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:34.284141 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:34.284629 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:34.284641 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:34.286428 (+ 1787us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41281: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":92,"threads_started":1}
W20260925 16:33:34.286698 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:33:34.281525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57881 (local address 127.0.98.62:41281)
0925 16:33:34.281674 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:34.281678 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:34.281819 (+ 141us) server_negotiation.cc:407] Connection header received
0925 16:33:34.281925 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:34.281927 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:34.281968 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:34.282041 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:34.282466 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:34.282998 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:34.283756 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:34.283964 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:34.284040 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:34.286572 (+ 2532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:57881: BlockingRecv error: recv got EOF from 127.0.98.2:57881 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 25 16:33:34 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:33:34 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:33:34.485539 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:34.480128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41281 (local address 127.0.98.3:33463)
0925 16:33:34.480413 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:34.480426 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:34.480570 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:34.480842 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:34.480845 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:34.481049 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:34.481165 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:34.481170 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:34.481926 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:34.481931 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:34.482481 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:34.482488 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:34.482995 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:34.482999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:34.483088 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:34.483592 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:34.483603 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:34.485347 (+ 1744us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41281: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":84,"threads_started":1}
W20260925 16:33:34.485584 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:33:34.480184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33463 (local address 127.0.98.62:41281)
0925 16:33:34.480354 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:34.480359 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:34.480504 (+ 145us) server_negotiation.cc:407] Connection header received
0925 16:33:34.480672 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:34.480676 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:34.480730 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:34.480820 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:34.481285 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:34.481812 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:34.482656 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:34.482882 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:34.482993 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:34.485474 (+ 2481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33463: BlockingRecv error: recv got EOF from 127.0.98.3:33463 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 25 16:33:35 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:33:35 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:33:35.100368 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:35.095250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41281 (local address 127.0.98.1:58117)
0925 16:33:35.095514 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:35.095528 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:35.095618 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:35.095862 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:35.095865 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:35.096044 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:35.096137 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:35.096142 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:35.096852 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:35.096855 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:35.097408 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:35.097415 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:35.097819 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:35.097821 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:35.097904 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:35.098401 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:35.098412 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:35.100233 (+ 1821us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41281: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":82,"threads_started":1}
W20260925 16:33:35.100513 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:33:35.095319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58117 (local address 127.0.98.62:41281)
0925 16:33:35.095561 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:35.095564 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:35.095577 (+ 13us) server_negotiation.cc:407] Connection header received
0925 16:33:35.095710 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:35.095712 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:35.095753 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:35.095830 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:35.096236 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:35.096764 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:35.097511 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:35.097721 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:35.097816 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:35.100371 (+ 2555us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58117: BlockingRecv error: recv got EOF from 127.0.98.1:58117 (error 108)
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":60,"threads_started":1}
Sep 25 16:33:35 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:33:35 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:33:35.293192 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:35.287554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41281 (local address 127.0.98.2:55869)
0925 16:33:35.287820 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:35.287833 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:35.287932 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:35.288184 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:35.288186 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:35.288366 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:35.288461 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:35.288466 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:35.289224 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:35.289226 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:35.289761 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:35.289767 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:35.290355 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:35.290359 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:35.290451 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:35.291090 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:35.291104 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:35.293065 (+ 1961us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41281: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":90,"threads_started":1}
W20260925 16:33:35.293413 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:33:35.287620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55869 (local address 127.0.98.62:41281)
0925 16:33:35.287785 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:35.287789 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:35.287910 (+ 121us) server_negotiation.cc:407] Connection header received
0925 16:33:35.288003 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:35.288005 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:35.288048 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:35.288140 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:35.288567 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:35.289125 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:35.289872 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:35.290219 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:35.290323 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:35.293187 (+ 2864us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55869: BlockingRecv error: recv got EOF from 127.0.98.2:55869 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 25 16:33:35 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:33:35 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:33:35.492285 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:35.486473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41281 (local address 127.0.98.3:53183)
0925 16:33:35.486746 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:35.486767 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:35.486864 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:35.487104 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:35.487107 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:35.487278 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:35.487380 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:35.487385 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:35.488244 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:35.488248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:35.489055 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:35.489069 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:35.489624 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:35.489627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:35.489694 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:35.490165 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:35.490176 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:35.492170 (+ 1994us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41281: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":135,"threads_started":1}
W20260925 16:33:35.492406 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:33:35.486752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53183 (local address 127.0.98.62:41281)
0925 16:33:35.486884 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:35.486887 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:35.486901 (+ 14us) server_negotiation.cc:407] Connection header received
0925 16:33:35.486941 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:35.486943 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:35.486986 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:35.487071 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:35.487483 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:35.488117 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:35.489245 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:35.489518 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:35.489601 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:35.492288 (+ 2687us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:53183: BlockingRecv error: recv got EOF from 127.0.98.3:53183 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 25 16:33:36 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:33:36 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:33:36.112263 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:36.101266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41281 (local address 127.0.98.1:57259)
0925 16:33:36.101602 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:36.101615 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:36.101708 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:36.102012 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:36.102014 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:36.102192 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:36.102288 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:36.102293 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:36.103105 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:36.103108 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:36.103668 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:36.103675 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:36.104212 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:36.104216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:36.104287 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:36.109998 (+ 5711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:36.110012 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:36.112117 (+ 2105us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41281: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":127,"threads_started":1}
W20260925 16:33:36.112536 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:33:36.101323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57259 (local address 127.0.98.62:41281)
0925 16:33:36.101601 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:36.101604 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:36.101687 (+ 83us) server_negotiation.cc:407] Connection header received
0925 16:33:36.101789 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:36.101792 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:36.101882 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:36.101972 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:36.102394 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:36.102988 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:36.103832 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:36.104075 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:36.104165 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:36.112278 (+ 8113us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:57259: BlockingRecv error: recv got EOF from 127.0.98.1:57259 (error 108)
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":80,"threads_started":1}
Sep 25 16:33:36 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:33:36 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:33:36.300161 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:36.294044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41281 (local address 127.0.98.2:32823)
0925 16:33:36.294318 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:36.294332 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:36.294433 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:36.294701 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:36.294703 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:36.294879 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:36.294974 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:36.294979 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:36.295839 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:36.295846 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:36.296621 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:36.296631 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:36.297179 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:36.297184 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:36.297278 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:36.297916 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:36.297931 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:36.299957 (+ 2026us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41281: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":92,"threads_started":1}
W20260925 16:33:36.300217 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:33:36.294143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32823 (local address 127.0.98.62:41281)
0925 16:33:36.294328 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:36.294332 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:36.294389 (+ 57us) server_negotiation.cc:407] Connection header received
0925 16:33:36.294514 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:36.294516 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:36.294562 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:36.294668 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:36.295095 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:36.295701 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:36.296759 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:36.297038 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:36.297161 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:36.300101 (+ 2940us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:32823: BlockingRecv error: recv got EOF from 127.0.98.2:32823 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 25 16:33:36 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:33:36 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:33:36.499337 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:36.493186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41281 (local address 127.0.98.3:46471)
0925 16:33:36.493489 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:36.493505 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:36.493616 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:36.493887 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:36.493889 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:36.494085 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:36.494230 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:36.494239 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:36.495438 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:36.495442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:36.496189 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:36.496200 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:36.496628 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:36.496631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:36.496696 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:36.497205 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:36.497217 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:36.499203 (+ 1986us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41281: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":89,"threads_started":1}
W20260925 16:33:36.499476 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:33:36.493362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46471 (local address 127.0.98.62:41281)
0925 16:33:36.493552 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:36.493556 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:36.493572 (+ 16us) server_negotiation.cc:407] Connection header received
0925 16:33:36.493714 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:36.493716 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:36.493762 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:36.493835 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:36.494407 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:36.495273 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:36.496307 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:36.496524 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:36.496603 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:36.499355 (+ 2752us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46471: BlockingRecv error: recv got EOF from 127.0.98.3:46471 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Sep 25 16:33:37 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:33:37 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:33:37.118851 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:37.113242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41281 (local address 127.0.98.1:55533)
0925 16:33:37.113635 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:37.113648 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:37.113738 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:37.113951 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:37.113953 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:37.114125 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:37.114213 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:37.114218 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:37.115016 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:37.115019 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:37.115601 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:37.115632 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:37.116062 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:37.116065 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:37.116126 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:37.116684 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:37.116695 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:37.118722 (+ 2027us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41281: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":91,"threads_started":1}
W20260925 16:33:37.119163 9588 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:33:37.113302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55533 (local address 127.0.98.62:41281)
0925 16:33:37.113610 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:37.113615 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:37.113702 (+ 87us) server_negotiation.cc:407] Connection header received
0925 16:33:37.113800 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:37.113802 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:37.113844 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:37.113917 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:37.114342 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:37.114902 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:37.115732 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:37.115965 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:37.116047 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:37.118885 (+ 2838us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:55533: BlockingRecv error: recv got EOF from 127.0.98.1:55533 (error 108)
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":112,"threads_started":1}
Sep 25 16:33:37 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:33:37 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:33:37.306905 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:37.301037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41281 (local address 127.0.98.2:58045)
0925 16:33:37.301433 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:37.301454 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:37.301592 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:37.301942 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:37.301949 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:37.302236 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:37.302398 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:37.302406 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:37.303250 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:37.303253 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:37.303811 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:37.303819 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:37.304308 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:37.304311 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:37.304368 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:37.304929 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:37.304940 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:37.306780 (+ 1840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41281: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":103,"threads_started":1}
W20260925 16:33:37.307073 9588 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:33:37.301208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58045 (local address 127.0.98.62:41281)
0925 16:33:37.301345 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:37.301349 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:37.301522 (+ 173us) server_negotiation.cc:407] Connection header received
0925 16:33:37.301716 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:37.301722 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:37.301792 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:37.301887 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:37.302512 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:37.303156 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:37.303944 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:37.304198 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:37.304303 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:37.306947 (+ 2644us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58045: BlockingRecv error: recv got EOF from 127.0.98.2:58045 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 25 16:33:37 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:33:37 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:33:37.505836 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:37.500199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41281 (local address 127.0.98.3:48155)
0925 16:33:37.500493 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:37.500506 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:37.500634 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:37.500912 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:37.500915 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:37.501131 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:37.501223 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:37.501228 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:37.501986 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:37.501992 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:37.502592 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:37.502630 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:37.503098 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:37.503101 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:37.503185 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:37.503739 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:37.503760 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:37.505623 (+ 1863us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41281: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":76,"threads_started":1}
W20260925 16:33:37.505854 9588 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:33:37.500271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48155 (local address 127.0.98.62:41281)
0925 16:33:37.500418 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:37.500422 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:37.500581 (+ 159us) server_negotiation.cc:407] Connection header received
0925 16:33:37.500739 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:37.500742 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:37.500791 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:37.500892 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:37.501353 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:37.501876 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:37.502743 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:37.502983 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:37.503096 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:37.505748 (+ 2652us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:48155: BlockingRecv error: recv got EOF from 127.0.98.3:48155 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 25 16:33:38 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:33:38 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:33:38.125322 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:38.119803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41281 (local address 127.0.98.1:38139)
0925 16:33:38.120128 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:38.120141 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:38.120275 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:38.120594 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:38.120597 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:38.120849 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:38.120972 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:38.120979 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:38.121806 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:38.121809 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:38.122372 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:38.122379 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:38.122845 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:38.122848 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:38.122914 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:38.123440 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:38.123451 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:38.125198 (+ 1747us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41281: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":81,"threads_started":1}
W20260925 16:33:38.125461 9592 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:33:38.119888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38139 (local address 127.0.98.62:41281)
0925 16:33:38.120138 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:38.120142 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:38.120239 (+ 97us) server_negotiation.cc:407] Connection header received
0925 16:33:38.120380 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:38.120386 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:38.120443 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:38.120569 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:38.121126 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:38.121705 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:38.122486 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:38.122723 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:38.122837 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:38.125331 (+ 2494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38139: BlockingRecv error: recv got EOF from 127.0.98.1:38139 (error 108)
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":59,"threads_started":1}
Sep 25 16:33:38 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:33:38 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:33:38.313306 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:38.307750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41281 (local address 127.0.98.2:57231)
0925 16:33:38.308070 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:38.308083 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:38.308174 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:38.308394 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:38.308396 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:38.308568 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:38.308663 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:38.308669 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:38.309445 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:38.309447 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:38.310009 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:38.310017 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:38.310461 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:38.310464 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:38.310539 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:38.311137 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:38.311148 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:38.313184 (+ 2036us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41281: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":143,"threads_started":1}
W20260925 16:33:38.313514 9592 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:33:38.307876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57231 (local address 127.0.98.62:41281)
0925 16:33:38.308036 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:38.308042 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:38.308134 (+ 92us) server_negotiation.cc:407] Connection header received
0925 16:33:38.308242 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:38.308244 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:38.308287 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:38.308360 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:38.308787 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:38.309353 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:38.310126 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:38.310369 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:38.310447 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:38.313321 (+ 2874us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:57231: BlockingRecv error: recv got EOF from 127.0.98.2:57231 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 25 16:33:38 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:33:38 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:33:38.512264 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:38.506886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41281 (local address 127.0.98.3:54569)
0925 16:33:38.507151 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:38.507164 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:38.507257 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:38.507485 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:38.507487 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:38.507660 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:38.507751 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:38.507756 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:38.508452 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:38.508454 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:38.509008 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:38.509016 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:38.509434 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:38.509436 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:38.509494 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:38.509994 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:38.510005 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:38.512046 (+ 2041us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41281: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":108,"threads_started":1}
W20260925 16:33:38.512286 9592 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:33:38.506950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54569 (local address 127.0.98.62:41281)
0925 16:33:38.507097 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:38.507101 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:38.507232 (+ 131us) server_negotiation.cc:407] Connection header received
0925 16:33:38.507333 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:38.507336 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:38.507383 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:38.507454 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:38.507846 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:38.508357 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:38.509113 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:38.509335 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:38.509414 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:38.512181 (+ 2767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54569: BlockingRecv error: recv got EOF from 127.0.98.3:54569 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 25 16:33:39 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:33:39 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:33:39.132454 9595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:39.126348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41281 (local address 127.0.98.1:45271)
0925 16:33:39.126665 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:39.126680 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:39.126775 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:39.127125 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:39.127129 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:39.127324 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:39.127429 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:39.127435 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:39.128427 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:39.128431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:39.129256 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:39.129266 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:39.129680 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:39.129683 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:39.129839 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:39.130367 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:39.130380 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:39.132338 (+ 1958us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41281: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":118,"threads_started":1}
W20260925 16:33:39.132612 9596 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:33:39.126429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45271 (local address 127.0.98.62:41281)
0925 16:33:39.126725 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:39.126729 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:39.126743 (+ 14us) server_negotiation.cc:407] Connection header received
0925 16:33:39.126861 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:39.126863 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:39.126907 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:39.126992 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:39.127551 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:39.128268 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:39.129365 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:39.129582 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:39.129662 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:39.132475 (+ 2813us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45271: BlockingRecv error: recv got EOF from 127.0.98.1:45271 (error 108)
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":89,"threads_started":1}
Sep 25 16:33:39 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:33:39 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:33:39.319468 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:39.314027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41281 (local address 127.0.98.2:53673)
0925 16:33:39.314293 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:39.314307 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:39.314405 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:39.314675 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:39.314677 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:39.314898 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:39.314995 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:39.315001 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:39.315789 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:39.315791 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:39.316345 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:39.316353 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:39.316820 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:39.316824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:39.316895 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:39.317470 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:39.317481 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:39.319300 (+ 1819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41281: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":66,"threads_started":1}
W20260925 16:33:39.319545 9596 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:33:39.314141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53673 (local address 127.0.98.62:41281)
0925 16:33:39.314304 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:39.314307 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:39.314378 (+ 71us) server_negotiation.cc:407] Connection header received
0925 16:33:39.314483 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:39.314485 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:39.314529 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:39.314659 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:39.315092 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:39.315657 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:39.316442 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:39.316695 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:39.316772 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:39.319430 (+ 2658us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53673: BlockingRecv error: recv got EOF from 127.0.98.2:53673 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 25 16:33:39 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:33:39 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:33:39.518747 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:39.513028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41281 (local address 127.0.98.3:56459)
0925 16:33:39.513361 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:39.513383 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:39.513525 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:39.513798 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:39.513800 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:39.514008 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:39.514096 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:39.514101 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:39.514967 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:39.514973 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:39.515602 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:39.515611 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:39.516089 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:39.516093 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:39.516175 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:39.516766 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:39.516777 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:39.518555 (+ 1778us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41281: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":83,"threads_started":1}
W20260925 16:33:39.518849 9596 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:33:39.513140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56459 (local address 127.0.98.62:41281)
0925 16:33:39.513316 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:39.513320 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:39.513461 (+ 141us) server_negotiation.cc:407] Connection header received
0925 16:33:39.513626 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:39.513630 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:39.513683 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:39.513777 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:39.514220 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:39.514859 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:39.515749 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:39.515980 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:39.516087 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:39.518700 (+ 2613us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56459: BlockingRecv error: recv got EOF from 127.0.98.3:56459 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 25 16:33:40 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:33:40 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:33:40.139206 9599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:40.133201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41281 (local address 127.0.98.1:44465)
0925 16:33:40.133565 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:40.133580 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:40.133723 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:40.133985 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:40.133987 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:40.134177 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:40.134277 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:40.134284 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:40.135165 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:40.135169 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:40.136141 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:40.136156 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:40.136621 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:40.136625 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:40.136717 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:40.137223 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:40.137234 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:40.139079 (+ 1845us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41281: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":98,"threads_started":1}
W20260925 16:33:40.139345 9600 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:33:40.133327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44465 (local address 127.0.98.62:41281)
0925 16:33:40.133570 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:40.133573 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:40.133690 (+ 117us) server_negotiation.cc:407] Connection header received
0925 16:33:40.133817 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:40.133819 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:40.133864 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:40.133942 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:40.134379 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:40.135021 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:40.136252 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:40.136500 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:40.136582 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:40.139214 (+ 2632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:44465: BlockingRecv error: recv got EOF from 127.0.98.1:44465 (error 108)
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":59,"threads_started":1}
Sep 25 16:33:40 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:33:40 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:33:40.325767 9601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:40.320271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41281 (local address 127.0.98.2:43321)
0925 16:33:40.320717 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:40.320740 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:40.320871 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:40.321176 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:40.321179 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:40.321431 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:40.321545 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:40.321552 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:40.322291 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:40.322293 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:40.322850 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:40.322857 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:40.323229 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:40.323231 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:40.323285 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:40.323843 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:40.323856 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:40.325649 (+ 1793us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41281: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":96,"threads_started":1}
W20260925 16:33:40.325925 9600 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:33:40.320351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43321 (local address 127.0.98.62:41281)
0925 16:33:40.320515 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:40.320521 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:40.320816 (+ 295us) server_negotiation.cc:407] Connection header received
0925 16:33:40.320992 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:40.320995 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:40.321052 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:40.321133 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:40.321656 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:40.322205 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:40.322938 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:40.323139 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:40.323213 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:40.325771 (+ 2558us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:43321: BlockingRecv error: recv got EOF from 127.0.98.2:43321 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 25 16:33:40 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:33:40 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:33:40.524943 9602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:40.519605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41281 (local address 127.0.98.3:54909)
0925 16:33:40.519899 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:40.519912 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:40.520005 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:40.520231 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:40.520233 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:40.520404 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:40.520495 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:40.520499 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:40.521226 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:40.521229 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:40.521840 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:40.521849 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:40.522248 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:40.522251 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:40.522333 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:40.522834 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:40.522847 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:40.524756 (+ 1909us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41281: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":86,"threads_started":1}
W20260925 16:33:40.525000 9600 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:33:40.519735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54909 (local address 127.0.98.62:41281)
0925 16:33:40.519897 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:40.519907 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:40.519981 (+ 74us) server_negotiation.cc:407] Connection header received
0925 16:33:40.520081 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:40.520083 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:40.520126 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:40.520196 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:40.520587 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:40.521093 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:40.521955 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:40.522154 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:40.522225 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:40.524886 (+ 2661us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54909: BlockingRecv error: recv got EOF from 127.0.98.3:54909 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 25 16:33:41 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:33:41 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:33:41.151003 9603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:41.140231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41281 (local address 127.0.98.1:40147)
0925 16:33:41.140574 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:41.140593 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:41.140712 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:41.140987 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:41.140995 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:41.141241 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:41.141393 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:41.141405 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:41.142284 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:41.142287 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:41.142923 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:41.142932 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:41.143370 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:41.143373 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:41.143446 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:41.148865 (+ 5419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:41.148878 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:41.150877 (+ 1999us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41281: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":119,"threads_started":1}
W20260925 16:33:41.151285 9604 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:33:41.140439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40147 (local address 127.0.98.62:41281)
0925 16:33:41.140703 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:41.140707 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:41.140722 (+ 15us) server_negotiation.cc:407] Connection header received
0925 16:33:41.140790 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:41.140794 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:41.140852 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:41.140927 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:41.141511 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:41.142156 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:41.143026 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:41.143237 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:41.143324 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:41.151016 (+ 7692us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:40147: BlockingRecv error: recv got EOF from 127.0.98.1:40147 (error 108)
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":100,"threads_started":1}
Sep 25 16:33:41 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:33:41 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:33:41.332363 9605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:41.326692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41281 (local address 127.0.98.2:53585)
0925 16:33:41.326935 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:41.326951 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:41.327074 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:41.327405 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:41.327408 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:41.327649 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:41.327746 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:41.327752 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:41.328674 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:41.328677 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:41.329243 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:41.329250 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:41.329699 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:41.329701 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:41.329761 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:41.330238 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:41.330249 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:41.332250 (+ 2001us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41281: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":79,"threads_started":1}
W20260925 16:33:41.332500 9604 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:33:41.326673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53585 (local address 127.0.98.62:41281)
0925 16:33:41.326859 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:41.326864 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:41.327041 (+ 177us) server_negotiation.cc:407] Connection header received
0925 16:33:41.327217 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:41.327220 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:41.327296 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:41.327386 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:41.327884 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:41.328559 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:41.329344 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:41.329592 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:41.329681 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:41.332385 (+ 2704us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53585: BlockingRecv error: recv got EOF from 127.0.98.2:53585 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 25 16:33:41 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:33:41 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:33:41.531652 9606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:41.525673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41281 (local address 127.0.98.3:47321)
0925 16:33:41.526024 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:41.526037 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:41.526164 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:41.526444 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:41.526447 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:41.526781 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:41.526896 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:41.526902 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:41.527638 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:41.527644 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:41.528322 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:41.528331 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:41.528795 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:41.528799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:41.528880 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:41.529499 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:41.529510 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:41.531541 (+ 2031us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41281: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":95,"threads_started":1}
W20260925 16:33:41.531810 9604 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:33:41.525789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47321 (local address 127.0.98.62:41281)
0925 16:33:41.525961 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:41.525967 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:41.526111 (+ 144us) server_negotiation.cc:407] Connection header received
0925 16:33:41.526271 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:41.526274 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:41.526339 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:41.526425 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:41.527005 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:41.527531 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:41.528451 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:41.528687 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:41.528792 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:41.531661 (+ 2869us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:47321: BlockingRecv error: recv got EOF from 127.0.98.3:47321 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 25 16:33:42 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:33:42 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:33:42.157531 9607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:42.151884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41281 (local address 127.0.98.1:55987)
0925 16:33:42.152237 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:42.152255 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:42.152365 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:42.152644 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:42.152647 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:42.152858 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:42.152977 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:42.152983 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:42.153780 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:42.153784 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:42.154347 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:42.154355 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:42.154867 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:42.154870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:42.154955 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:42.155448 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:42.155459 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:42.157363 (+ 1904us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41281: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":101,"threads_started":1}
W20260925 16:33:42.157605 9608 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:33:42.151917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55987 (local address 127.0.98.62:41281)
0925 16:33:42.152258 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:42.152263 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:42.152313 (+ 50us) server_negotiation.cc:407] Connection header received
0925 16:33:42.152474 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:42.152478 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:42.152529 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:42.152620 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:42.153114 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:42.153663 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:42.154465 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:42.154754 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:42.154863 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:42.157483 (+ 2620us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:55987: BlockingRecv error: recv got EOF from 127.0.98.1:55987 (error 108)
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":99,"threads_started":1}
Sep 25 16:33:42 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:33:42 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:33:42.338827 9609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:42.333132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41281 (local address 127.0.98.2:48551)
0925 16:33:42.333391 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:42.333405 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:42.333499 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:42.333737 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:42.333740 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:42.333920 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:42.334013 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:42.334018 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:42.334811 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:42.334813 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:42.335680 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:42.335690 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:42.336253 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:42.336255 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:42.336314 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:42.336802 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:42.336813 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:42.338707 (+ 1894us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41281: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":90,"threads_started":1}
W20260925 16:33:42.338948 9608 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:33:42.333217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48551 (local address 127.0.98.62:41281)
0925 16:33:42.333382 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:42.333385 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:42.333474 (+ 89us) server_negotiation.cc:407] Connection header received
0925 16:33:42.333579 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:42.333582 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:42.333628 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:42.333700 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:42.334114 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:42.334697 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:42.335792 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:42.336149 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:42.336229 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:42.338832 (+ 2603us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48551: BlockingRecv error: recv got EOF from 127.0.98.2:48551 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 25 16:33:42 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:33:42 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:33:42.538157 9610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:42.532414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41281 (local address 127.0.98.3:60645)
0925 16:33:42.532761 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:42.532779 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:42.532926 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:42.533211 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:42.533214 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:42.533431 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:42.533519 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:42.533524 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:42.534288 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:42.534293 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:42.534904 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:42.534912 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:42.535468 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:42.535472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:42.535576 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:42.536205 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:42.536222 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:42.538039 (+ 1817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41281: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":87,"threads_started":1}
W20260925 16:33:42.538331 9608 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:33:42.532555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60645 (local address 127.0.98.62:41281)
0925 16:33:42.532738 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:42.532743 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:42.532867 (+ 124us) server_negotiation.cc:407] Connection header received
0925 16:33:42.533039 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:42.533043 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:42.533099 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:42.533187 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:42.533647 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:42.534179 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:42.535026 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:42.535328 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:42.535435 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:42.538175 (+ 2740us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:60645: BlockingRecv error: recv got EOF from 127.0.98.3:60645 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 25 16:33:43 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:33:43 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:33:43.163702 9611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:43.158447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41281 (local address 127.0.98.1:33873)
0925 16:33:43.158751 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:43.158765 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:43.158852 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:43.159108 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:43.159110 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:43.159280 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:43.159373 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:43.159379 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:43.160085 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:43.160087 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:43.160673 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:43.160682 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:43.161090 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:43.161093 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:43.161154 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:43.161715 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:43.161727 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:43.163588 (+ 1861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41281: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":84,"threads_started":1}
W20260925 16:33:43.163856 9612 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:33:43.158527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33873 (local address 127.0.98.62:41281)
0925 16:33:43.158789 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:43.158792 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:43.158810 (+ 18us) server_negotiation.cc:407] Connection header received
0925 16:33:43.158951 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:43.158953 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:43.158994 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:43.159070 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:43.159472 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:43.159983 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:43.160775 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:43.160988 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:43.161060 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:43.163725 (+ 2665us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33873: BlockingRecv error: recv got EOF from 127.0.98.1:33873 (error 108)
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":91,"threads_started":1}
Sep 25 16:33:43 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:33:43 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:33:43.345038 9613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:43.339745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41281 (local address 127.0.98.2:60061)
0925 16:33:43.340015 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:43.340030 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:43.340119 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:43.340348 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:43.340350 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:43.340550 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:43.340643 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:43.340648 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:43.341342 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:43.341344 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:43.341924 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:43.341933 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:43.342360 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:43.342363 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:43.342431 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:43.342979 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:43.342991 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:43.344866 (+ 1875us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41281: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":101,"threads_started":1}
W20260925 16:33:43.345086 9612 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:33:43.339794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60061 (local address 127.0.98.62:41281)
0925 16:33:43.339940 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:43.339943 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:43.340098 (+ 155us) server_negotiation.cc:407] Connection header received
0925 16:33:43.340206 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:43.340208 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:43.340247 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:43.340313 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:43.340740 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:43.341252 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:43.342026 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:43.342240 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:43.342314 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:43.344985 (+ 2671us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:60061: BlockingRecv error: recv got EOF from 127.0.98.2:60061 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 25 16:33:43 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:33:43 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:33:43.544896 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:43.538936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41281 (local address 127.0.98.3:60487)
0925 16:33:43.539288 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:43.539309 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:43.539475 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:43.539808 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:43.539812 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:43.540052 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:43.540157 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:43.540164 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:43.541063 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:43.541069 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:43.541709 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:43.541718 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:43.542273 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:43.542278 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:43.542381 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:43.542937 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:43.542951 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:43.544780 (+ 1829us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41281: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":92,"threads_started":1}
W20260925 16:33:43.545133 9612 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:33:43.539112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60487 (local address 127.0.98.62:41281)
0925 16:33:43.539292 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:43.539299 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:43.539412 (+ 113us) server_negotiation.cc:407] Connection header received
0925 16:33:43.539602 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:43.539607 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:43.539671 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:43.539788 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:43.540298 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:43.540907 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:43.541851 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:43.542144 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:43.542273 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:43.544967 (+ 2694us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:60487: BlockingRecv error: recv got EOF from 127.0.98.3:60487 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 25 16:33:44 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:33:44 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:33:44.169775 9615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:44.164535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41281 (local address 127.0.98.1:37801)
0925 16:33:44.164788 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:44.164802 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:44.164890 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:44.165130 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:44.165132 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:44.165308 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:44.165402 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:44.165406 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:44.166135 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:44.166139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:44.166744 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:44.166754 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:44.167159 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:44.167162 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:44.167224 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:44.167744 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:44.167755 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:44.169659 (+ 1904us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41281: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":88,"threads_started":1}
W20260925 16:33:44.169960 9616 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:33:44.164595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37801 (local address 127.0.98.62:41281)
0925 16:33:44.164857 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:44.164860 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:44.164876 (+ 16us) server_negotiation.cc:407] Connection header received
0925 16:33:44.164972 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:44.164975 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:44.165016 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:44.165099 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:44.165494 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:44.166007 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:44.166846 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:44.167060 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:44.167135 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:44.169790 (+ 2655us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37801: BlockingRecv error: recv got EOF from 127.0.98.1:37801 (error 108)
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":113,"threads_started":1}
Sep 25 16:33:44 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:33:44 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:33:44.351543 9617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:44.345897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41281 (local address 127.0.98.2:59483)
0925 16:33:44.346199 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:44.346225 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:44.346336 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:44.346641 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:44.346644 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:44.346818 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:44.346909 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:44.346914 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:44.347724 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:44.347726 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:44.348270 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:44.348277 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:44.348804 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:44.348807 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:44.348874 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:44.349389 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:44.349401 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:44.351384 (+ 1983us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41281: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":101,"threads_started":1}
W20260925 16:33:44.351613 9616 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:33:44.345994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59483 (local address 127.0.98.62:41281)
0925 16:33:44.346285 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:44.346292 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:44.346308 (+ 16us) server_negotiation.cc:407] Connection header received
0925 16:33:44.346484 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:44.346486 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:44.346529 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:44.346616 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:44.347041 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:44.347624 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:44.348384 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:44.348660 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:44.348753 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:44.351519 (+ 2766us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:59483: BlockingRecv error: recv got EOF from 127.0.98.2:59483 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 25 16:33:44 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:33:44 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:33:44.551025 9618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:44.545747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41281 (local address 127.0.98.3:43591)
0925 16:33:44.546098 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:44.546114 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:44.546264 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:44.546570 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:44.546573 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:44.546850 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:44.546944 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:44.546950 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:44.547652 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:44.547654 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:44.548187 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:44.548193 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:44.548578 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:44.548580 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:44.548649 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:44.549127 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:44.549138 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:44.550909 (+ 1771us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41281: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":95,"threads_started":1}
W20260925 16:33:44.551206 9616 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:33:44.545804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43591 (local address 127.0.98.62:41281)
0925 16:33:44.545951 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:44.545956 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:44.546226 (+ 270us) server_negotiation.cc:407] Connection header received
0925 16:33:44.546371 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:44.546375 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:44.546431 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:44.546517 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:44.547045 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:44.547562 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:44.548279 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:44.548488 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:44.548560 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:44.551065 (+ 2505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43591: BlockingRecv error: recv got EOF from 127.0.98.3:43591 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 25 16:33:45 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:33:45 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:33:45.176203 9619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:45.170518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41281 (local address 127.0.98.1:51517)
0925 16:33:45.170837 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:45.170851 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:45.170941 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:45.171184 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:45.171186 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:45.171358 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:45.171450 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:45.171455 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:45.172236 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:45.172239 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:45.172783 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:45.172790 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:45.173267 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:45.173269 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:45.173326 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:45.173966 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:45.173977 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:45.176084 (+ 2107us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41281: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":137,"threads_started":1}
W20260925 16:33:45.176362 9620 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:33:45.170668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51517 (local address 127.0.98.62:41281)
0925 16:33:45.170895 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:45.170898 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:45.170911 (+ 13us) server_negotiation.cc:407] Connection header received
0925 16:33:45.171032 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:45.171034 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:45.171077 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:45.171153 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:45.171586 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:45.172136 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:45.172924 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:45.173160 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:45.173256 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:45.176221 (+ 2965us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51517: BlockingRecv error: recv got EOF from 127.0.98.1:51517 (error 108)
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":60,"threads_started":1}
Sep 25 16:33:45 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:33:45 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:33:45.358207 9621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:45.352451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41281 (local address 127.0.98.2:54027)
0925 16:33:45.352776 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:45.352798 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:45.352923 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:45.353220 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:45.353223 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:45.353436 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:45.353528 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:45.353534 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:45.354450 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:45.354455 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:45.355128 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:45.355137 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:45.355605 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:45.355608 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:45.355700 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:45.356194 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:45.356205 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:45.358054 (+ 1849us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41281: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":107,"threads_started":1}
W20260925 16:33:45.358323 9620 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:33:45.352569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54027 (local address 127.0.98.62:41281)
0925 16:33:45.352726 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:45.352731 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:45.352872 (+ 141us) server_negotiation.cc:407] Connection header received
0925 16:33:45.353033 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:45.353037 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:45.353084 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:45.353195 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:45.353656 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:45.354340 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:45.355251 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:45.355491 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:45.355601 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:45.358193 (+ 2592us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54027: BlockingRecv error: recv got EOF from 127.0.98.2:54027 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 25 16:33:45 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:33:45 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:33:45.557826 9622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:45.552039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41281 (local address 127.0.98.3:43521)
0925 16:33:45.552321 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:45.552337 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:45.552479 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:45.552708 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:45.552710 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:45.552897 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:45.552997 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:45.553002 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:45.553755 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:45.553758 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:45.554325 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:45.554334 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:45.554874 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:45.554878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:45.554992 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:45.555676 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:45.555700 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:45.557711 (+ 2011us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41281: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":88,"threads_started":1}
W20260925 16:33:45.557957 9620 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:33:45.552110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43521 (local address 127.0.98.62:41281)
0925 16:33:45.552258 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:45.552264 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:45.552414 (+ 150us) server_negotiation.cc:407] Connection header received
0925 16:33:45.552580 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:45.552583 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:45.552626 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:45.552692 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:45.553094 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:45.553639 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:45.554440 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:45.554734 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:45.554843 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:45.557841 (+ 2998us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43521: BlockingRecv error: recv got EOF from 127.0.98.3:43521 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 25 16:33:46 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:33:46 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:33:46.188932 9623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:46.177098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41281 (local address 127.0.98.1:55495)
0925 16:33:46.177355 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:46.177368 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:46.177465 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:46.177703 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:46.177705 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:46.177876 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:46.177968 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:46.177973 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:46.178746 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:46.178749 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:46.179300 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:46.179308 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:46.179720 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:46.179722 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:46.179779 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:46.186886 (+ 7107us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:46.186901 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:46.188806 (+ 1905us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41281: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":85,"threads_started":1}
W20260925 16:33:46.189164 9624 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:33:46.177158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55495 (local address 127.0.98.62:41281)
0925 16:33:46.177381 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:46.177391 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:46.177418 (+ 27us) server_negotiation.cc:407] Connection header received
0925 16:33:46.177549 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:46.177551 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:46.177594 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:46.177669 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:46.178063 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:46.178646 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:46.179397 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:46.179625 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:46.179702 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:46.189012 (+ 9310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:55495: BlockingRecv error: recv got EOF from 127.0.98.1:55495 (error 108)
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":58,"threads_started":1}
Sep 25 16:33:46 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:33:46 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:33:46.365839 9625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:46.359186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41281 (local address 127.0.98.2:47165)
0925 16:33:46.359580 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:46.359597 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:46.359715 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:46.360018 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:46.360021 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:46.360291 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:46.360404 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:46.360410 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:46.361420 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:46.361426 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:46.362143 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:46.362155 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:46.362797 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:46.362801 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:46.362924 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:46.363553 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:46.363569 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:46.365719 (+ 2150us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41281: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":80,"threads_started":1}
W20260925 16:33:46.366202 9624 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:33:46.359349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47165 (local address 127.0.98.62:41281)
0925 16:33:46.359541 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:46.359547 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:46.359803 (+ 256us) server_negotiation.cc:407] Connection header received
0925 16:33:46.359844 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:46.359848 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:46.359895 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:46.360001 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:46.360516 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:46.361315 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:46.362282 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:46.362653 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:46.362764 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:46.365879 (+ 3115us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:47165: BlockingRecv error: recv got EOF from 127.0.98.2:47165 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 25 16:33:46 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:33:46 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:33:46.564340 9626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:46.558773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41281 (local address 127.0.98.3:39571)
0925 16:33:46.559088 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:46.559101 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:46.559225 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:46.559538 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:46.559541 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:46.559740 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:46.559825 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:46.559830 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:46.560568 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:46.560574 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:46.561140 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:46.561148 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:46.561594 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:46.561598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:46.561680 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:46.562204 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:46.562215 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:46.564140 (+ 1925us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41281: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":131,"threads_started":1}
W20260925 16:33:46.564396 9624 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:33:46.558838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39571 (local address 127.0.98.62:41281)
0925 16:33:46.559002 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:46.559007 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:46.559176 (+ 169us) server_negotiation.cc:407] Connection header received
0925 16:33:46.559330 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:46.559334 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:46.559391 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:46.559478 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:46.559947 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:46.560462 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:46.561268 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:46.561488 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:46.561592 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:46.564275 (+ 2683us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:39571: BlockingRecv error: recv got EOF from 127.0.98.3:39571 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 25 16:33:47 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:33:47 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:33:47.195012 9627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:47.189728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41281 (local address 127.0.98.1:46843)
0925 16:33:47.190004 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:47.190017 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:47.190113 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:47.190347 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:47.190349 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:47.190516 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:47.190669 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:47.190675 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:47.191373 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:47.191375 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:47.191930 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:47.191937 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:47.192342 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:47.192344 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:47.192401 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:47.192911 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:47.192921 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:47.194808 (+ 1887us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41281: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":105,"threads_started":1}
W20260925 16:33:47.195043 9628 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:33:47.189789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46843 (local address 127.0.98.62:41281)
0925 16:33:47.190087 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:47.190090 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:47.190103 (+ 13us) server_negotiation.cc:407] Connection header received
0925 16:33:47.190198 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:47.190200 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:47.190242 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:47.190314 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:47.190777 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:47.191285 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:47.192034 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:47.192249 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:47.192325 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:47.194925 (+ 2600us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46843: BlockingRecv error: recv got EOF from 127.0.98.1:46843 (error 108)
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":80,"threads_started":1}
Sep 25 16:33:47 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:33:47 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:33:47.372164 9629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:47.366729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41281 (local address 127.0.98.2:41901)
0925 16:33:47.367027 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:47.367041 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:47.367143 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:47.367363 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:47.367365 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:47.367568 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:47.367659 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:47.367664 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:47.368432 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:47.368435 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:47.368977 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:47.368984 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:47.369438 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:47.369440 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:47.369498 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:47.369980 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:47.369991 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:47.371978 (+ 1987us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41281: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":118,"threads_started":1}
W20260925 16:33:47.372197 9628 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:33:47.366763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41901 (local address 127.0.98.62:41281)
0925 16:33:47.366925 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:47.366929 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:47.367097 (+ 168us) server_negotiation.cc:407] Connection header received
0925 16:33:47.367210 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:47.367212 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:47.367255 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:47.367325 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:47.367760 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:47.368296 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:47.369079 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:47.369307 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:47.369394 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:47.372097 (+ 2703us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41901: BlockingRecv error: recv got EOF from 127.0.98.2:41901 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 25 16:33:47 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:33:47 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:33:47.570487 9630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:47.565141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41281 (local address 127.0.98.3:47457)
0925 16:33:47.565505 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:47.565519 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:47.565623 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:47.565873 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:47.565876 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:47.566071 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:47.566178 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:47.566183 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:47.566927 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:47.566930 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:47.567523 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:47.567531 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:47.567929 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:47.567931 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:47.567989 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:47.568490 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:47.568501 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:47.570365 (+ 1864us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41281: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":206,"threads_started":1}
W20260925 16:33:47.570668 9628 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:33:47.565249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47457 (local address 127.0.98.62:41281)
0925 16:33:47.565381 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:47.565384 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:47.565597 (+ 213us) server_negotiation.cc:407] Connection header received
0925 16:33:47.565692 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:47.565694 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:47.565736 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:47.565809 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:47.566285 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:47.566809 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:47.567626 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:47.567832 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:47.567904 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:47.570492 (+ 2588us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:47457: BlockingRecv error: recv got EOF from 127.0.98.3:47457 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 25 16:33:48 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:33:48 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:33:48.201318 9631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:48.195879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41281 (local address 127.0.98.1:51947)
0925 16:33:48.196191 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:48.196209 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:48.196311 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:48.196553 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:48.196556 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:48.196727 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:48.196830 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:48.196835 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:48.197608 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:48.197611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:48.198151 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:48.198158 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:48.198675 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:48.198678 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:48.198741 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:48.199307 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:48.199318 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:48.201181 (+ 1863us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41281: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":67,"threads_started":1}
W20260925 16:33:48.201483 9632 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:33:48.196077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51947 (local address 127.0.98.62:41281)
0925 16:33:48.196316 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:48.196319 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:48.196333 (+ 14us) server_negotiation.cc:407] Connection header received
0925 16:33:48.196388 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:48.196390 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:48.196431 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:48.196505 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:48.196931 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:48.197517 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:48.198258 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:48.198518 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:48.198631 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:48.201313 (+ 2682us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51947: BlockingRecv error: recv got EOF from 127.0.98.1:51947 (error 108)
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":70,"threads_started":1}
Sep 25 16:33:48 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:33:48 dist-test-slave-65pf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:33:48.378484 9633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:48.373130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41281 (local address 127.0.98.2:34601)
0925 16:33:48.373384 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:48.373397 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:48.373486 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:48.373712 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:48.373714 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:48.373883 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:48.373974 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:48.373979 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:48.374799 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:48.374803 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:48.375420 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:48.375429 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:48.375908 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:48.375912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:48.375995 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:48.376494 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:48.376506 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:48.378287 (+ 1781us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41281: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":74,"threads_started":1}
W20260925 16:33:48.378525 9632 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:33:48.373197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34601 (local address 127.0.98.62:41281)
0925 16:33:48.373333 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:48.373337 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:48.373448 (+ 111us) server_negotiation.cc:407] Connection header received
0925 16:33:48.373572 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:48.373574 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:48.373614 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:48.373681 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:48.374077 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:48.374677 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:48.375548 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:48.375781 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:48.375893 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:48.378421 (+ 2528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34601: BlockingRecv error: recv got EOF from 127.0.98.2:34601 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 16:33:48.473502 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 9156
I20260925 16:33:48.479296 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 9308
I20260925 16:33:48.484560 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 9440
I20260925 16:33:48.489997 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 9086
2026-09-25T16:33:48Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17941 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-25T16:33:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:33:48Z Disabled control of system clock
I20260925 16:33:48.520334 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353966814071-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353966814071-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353966814071-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45361
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45361
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 16:33:48.630172 9637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:33:48.630461 9637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:33:48.630497 9637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:33:48.634616 9637 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:33:48.634689 9637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:33:48.634704 9637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:33:48.634721 9637 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:33:48.634737 9637 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:33:48.639649 9637 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353966814071-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353966814071-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45361
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353966814071-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:33:48.640821 9637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:33:48.642261 9637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 16:33:48.651254 9637 server_base.cc:1034] running on GCE node
W20260925 16:33:48.651297 9645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:48.651298 9643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:48.651278 9642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:48.652064 9637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:33:48.653164 9637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:33:48.654381 9637 hybrid_clock.cc:648] HybridClock initialized: now 1790354028654339 us; error 32 us; skew 500 ppm
I20260925 16:33:48.656374 9637 webserver.cc:460] Webserver started at http://127.0.98.62:41439/ using document root <none> and password file <none>
I20260925 16:33:48.656970 9637 fs_manager.cc:359] Metadata directory not provided
I20260925 16:33:48.657063 9637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:33:48.657325 9637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:33:48.659451 9637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353966814071-392-0/minicluster-data/master-0/data/instance:
uuid: "1825034dd1b9414b86fb0fbaa62e8821"
format_stamp: "Formatted at 2026-09-25 16:33:48 on dist-test-slave-65pf"
server_key: "97134bce02fd549bd119dc0b4a8ec3d1"
server_key_iv: "44ff68721408a969c058355f3ba539d7"
server_key_version: "encryptionkey@0"
I20260925 16:33:48.660012 9637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353966814071-392-0/minicluster-data/master-0/wal/instance:
uuid: "1825034dd1b9414b86fb0fbaa62e8821"
format_stamp: "Formatted at 2026-09-25 16:33:48 on dist-test-slave-65pf"
server_key: "97134bce02fd549bd119dc0b4a8ec3d1"
server_key_iv: "44ff68721408a969c058355f3ba539d7"
server_key_version: "encryptionkey@0"
I20260925 16:33:48.663590 9637 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 16:33:48.665979 9651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:48.667362 9637 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:33:48.667519 9637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353966814071-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353966814071-392-0/minicluster-data/master-0/wal
uuid: "1825034dd1b9414b86fb0fbaa62e8821"
format_stamp: "Formatted at 2026-09-25 16:33:48 on dist-test-slave-65pf"
server_key: "97134bce02fd549bd119dc0b4a8ec3d1"
server_key_iv: "44ff68721408a969c058355f3ba539d7"
server_key_version: "encryptionkey@0"
I20260925 16:33:48.667636 9637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353966814071-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353966814071-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353966814071-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:33:48.686733 9637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:33:48.687431 9637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:33:48.687641 9637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:33:48.694317 9637 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45361
I20260925 16:33:48.694366 9703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45361 every 8 connection(s)
I20260925 16:33:48.695346 9637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:33:48.696056 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 9637
I20260925 16:33:48.696185 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353966814071-392-0/minicluster-data/master-0/wal/instance
I20260925 16:33:48.696511 392 external_mini_cluster.cc:1428] Setting key bd3961e428d77eb1fb33f62160a4e9fb
I20260925 16:33:48.699054 9704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:33:48.704339 9704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1825034dd1b9414b86fb0fbaa62e8821: Bootstrap starting.
I20260925 16:33:48.705686 9707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:48.697903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52156 (local address 127.0.98.62:45361)
0925 16:33:48.698512 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:48.698527 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:48.698562 (+ 35us) server_negotiation.cc:407] Connection header received
0925 16:33:48.699338 (+ 776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:48.699355 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:48.699744 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:48.700107 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:48.700521 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:48.701481 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:48.702360 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:48.702896 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:48.703079 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:48.703968 (+ 889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:48.703973 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:48.703985 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:33:48.704366 (+ 381us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:48.704417 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:33:48.704436 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:48.704553 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:48.704715 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 16:33:48.704998 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":126,"threads_started":1}
I20260925 16:33:48.706688 9704 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1825034dd1b9414b86fb0fbaa62e8821: Neither blocks nor log segments found. Creating new log.
I20260925 16:33:48.707420 9704 log.cc:824] T 00000000000000000000000000000000 P 1825034dd1b9414b86fb0fbaa62e8821: Log is configured to *not* fsync() on all Append() calls
I20260925 16:33:48.709203 9704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1825034dd1b9414b86fb0fbaa62e8821: No bootstrap required, opened a new log
I20260925 16:33:48.712781 9704 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1825034dd1b9414b86fb0fbaa62e8821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1825034dd1b9414b86fb0fbaa62e8821" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45361 } }
I20260925 16:33:48.712987 9704 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1825034dd1b9414b86fb0fbaa62e8821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:33:48.713060 9704 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1825034dd1b9414b86fb0fbaa62e8821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1825034dd1b9414b86fb0fbaa62e8821, State: Initialized, Role: FOLLOWER
I20260925 16:33:48.713450 9704 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1825034dd1b9414b86fb0fbaa62e8821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1825034dd1b9414b86fb0fbaa62e8821" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45361 } }
I20260925 16:33:48.713563 9704 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1825034dd1b9414b86fb0fbaa62e8821 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:33:48.713644 9704 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1825034dd1b9414b86fb0fbaa62e8821 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:33:48.713742 9704 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1825034dd1b9414b86fb0fbaa62e8821 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:33:48.714644 9704 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1825034dd1b9414b86fb0fbaa62e8821 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1825034dd1b9414b86fb0fbaa62e8821" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45361 } }
I20260925 16:33:48.714977 9704 leader_election.cc:302] T 00000000000000000000000000000000 P 1825034dd1b9414b86fb0fbaa62e8821 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1825034dd1b9414b86fb0fbaa62e8821; no voters:
I20260925 16:33:48.715262 9704 leader_election.cc:288] T 00000000000000000000000000000000 P 1825034dd1b9414b86fb0fbaa62e8821 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:33:48.715389 9709 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1825034dd1b9414b86fb0fbaa62e8821 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:33:48.715633 9709 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1825034dd1b9414b86fb0fbaa62e8821 [term 1 LEADER]: Becoming Leader. State: Replica: 1825034dd1b9414b86fb0fbaa62e8821, State: Running, Role: LEADER
I20260925 16:33:48.715941 9709 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1825034dd1b9414b86fb0fbaa62e8821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1825034dd1b9414b86fb0fbaa62e8821" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45361 } }
I20260925 16:33:48.716678 9704 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1825034dd1b9414b86fb0fbaa62e8821 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:33:48.717541 9710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1825034dd1b9414b86fb0fbaa62e8821 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1825034dd1b9414b86fb0fbaa62e8821. Latest consensus state: current_term: 1 leader_uuid: "1825034dd1b9414b86fb0fbaa62e8821" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1825034dd1b9414b86fb0fbaa62e8821" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45361 } } }
I20260925 16:33:48.717587 9711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1825034dd1b9414b86fb0fbaa62e8821 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1825034dd1b9414b86fb0fbaa62e8821" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1825034dd1b9414b86fb0fbaa62e8821" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45361 } } }
I20260925 16:33:48.717824 9710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1825034dd1b9414b86fb0fbaa62e8821 [sys.catalog]: This master's current role is: LEADER
I20260925 16:33:48.717845 9711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1825034dd1b9414b86fb0fbaa62e8821 [sys.catalog]: This master's current role is: LEADER
I20260925 16:33:48.718976 9718 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:33:48.721624 9718 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:33:48.727552 9718 catalog_manager.cc:1347] Generated new cluster ID: cd5bcd7ff34e495380fab63cc21f6590
I20260925 16:33:48.727630 9718 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:33:48.738353 9718 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:33:48.739595 9718 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:33:48.745671 9718 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1825034dd1b9414b86fb0fbaa62e8821: Generated new TSK 0
I20260925 16:33:48.746465 9718 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:33:48.751495 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353966814071-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353966814071-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353966814071-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45361
--builtin_ntp_servers=127.0.98.20:37315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 16:33:48.863969 9728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:33:48.864251 9728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:33:48.864290 9728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:33:48.868471 9728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:33:48.868546 9728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:33:48.868602 9728 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 16:33:48.873680 9728 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353966814071-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353966814071-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353966814071-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:33:48.874866 9728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:33:48.876312 9728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:33:48.886133 9734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:48.886149 9733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:48.886216 9736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:48.886358 9728 server_base.cc:1034] running on GCE node
I20260925 16:33:48.886890 9728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:33:48.887566 9728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:33:48.888758 9728 hybrid_clock.cc:648] HybridClock initialized: now 1790354028888737 us; error 35 us; skew 500 ppm
I20260925 16:33:48.890683 9728 webserver.cc:460] Webserver started at http://127.0.98.1:46213/ using document root <none> and password file <none>
I20260925 16:33:48.891269 9728 fs_manager.cc:359] Metadata directory not provided
I20260925 16:33:48.891362 9728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:33:48.891605 9728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:33:48.893685 9728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353966814071-392-0/minicluster-data/ts-0/data/instance:
uuid: "2f57296a426845f08852474d94f9f6a4"
format_stamp: "Formatted at 2026-09-25 16:33:48 on dist-test-slave-65pf"
server_key: "28bb60fe129c0a83d2f43b0df96631d0"
server_key_iv: "3dabed036fd0d18dd3baeece7f2ecd61"
server_key_version: "encryptionkey@0"
I20260925 16:33:48.894243 9728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353966814071-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2f57296a426845f08852474d94f9f6a4"
format_stamp: "Formatted at 2026-09-25 16:33:48 on dist-test-slave-65pf"
server_key: "28bb60fe129c0a83d2f43b0df96631d0"
server_key_iv: "3dabed036fd0d18dd3baeece7f2ecd61"
server_key_version: "encryptionkey@0"
I20260925 16:33:48.897943 9728 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 16:33:48.900599 9742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:48.901870 9728 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:33:48.902009 9728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353966814071-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353966814071-392-0/minicluster-data/ts-0/wal
uuid: "2f57296a426845f08852474d94f9f6a4"
format_stamp: "Formatted at 2026-09-25 16:33:48 on dist-test-slave-65pf"
server_key: "28bb60fe129c0a83d2f43b0df96631d0"
server_key_iv: "3dabed036fd0d18dd3baeece7f2ecd61"
server_key_version: "encryptionkey@0"
I20260925 16:33:48.902130 9728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353966814071-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353966814071-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353966814071-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:33:48.920655 9728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:33:48.921360 9728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:33:48.921566 9728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:33:48.922150 9728 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:33:48.923156 9728 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:33:48.923228 9728 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:48.923300 9728 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:33:48.923352 9728 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:48.932408 9728 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46309
I20260925 16:33:48.932430 9855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46309 every 8 connection(s)
I20260925 16:33:48.933322 9728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
I20260925 16:33:48.938382 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 9728
I20260925 16:33:48.938516 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353966814071-392-0/minicluster-data/ts-0/wal/instance
I20260925 16:33:48.938876 392 external_mini_cluster.cc:1428] Setting key 02914ad438b620a9f8de1127d34c1bfa
I20260925 16:33:48.940814 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353966814071-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353966814071-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353966814071-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45361
--builtin_ntp_servers=127.0.98.20:37315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 16:33:48.944705 9707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:48.935195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43461 (local address 127.0.98.62:45361)
0925 16:33:48.935367 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:48.935373 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:48.937009 (+ 1636us) server_negotiation.cc:407] Connection header received
0925 16:33:48.938839 (+ 1830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:48.938843 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:48.938933 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:48.939157 (+ 224us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:48.940657 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:48.941477 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:48.942710 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:48.942939 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:48.943056 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:48.944182 (+ 1126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:48.944187 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:48.944195 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:33:48.944207 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:48.944225 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:33:48.944233 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:48.944310 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:48.944482 (+ 172us) server_negotiation.cc:299] Negotiation successful
0925 16:33:48.944563 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 16:33:48.945564 9858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:48.935668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45361 (local address 127.0.98.1:43461)
0925 16:33:48.936792 (+ 1124us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:48.936846 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:48.938552 (+ 1706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:48.939799 (+ 1247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:48.939807 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:48.939904 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:48.940510 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:48.940525 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:48.941612 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:48.941615 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:48.942511 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:48.942520 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:48.943099 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:48.943103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:48.943270 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:48.943921 (+ 651us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 16:33:48.943923 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:48.943964 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 16:33:48.944029 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:33:48.944323 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:33:48.944333 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:33:48.944339 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 16:33:48.944473 (+ 134us) client_negotiation.cc:240] Negotiation successful
0925 16:33:48.944762 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":159,"threads_started":1}
I20260925 16:33:48.946911 9856 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45361
I20260925 16:33:48.947163 9856 heartbeater.cc:464] Registering TS with master...
I20260925 16:33:48.947688 9856 heartbeater.cc:511] Master 127.0.98.62:45361 requested a full tablet report, sending...
I20260925 16:33:48.949399 9668 ts_manager.cc:194] Registered new tserver with Master: 2f57296a426845f08852474d94f9f6a4 (127.0.98.1:46309)
I20260925 16:33:48.951215 9668 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:43461
W20260925 16:33:49.054678 9859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:33:49.054925 9859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:33:49.054961 9859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:33:49.059095 9859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:33:49.059149 9859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:33:49.059201 9859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 16:33:49.063762 9859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353966814071-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353966814071-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353966814071-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:33:49.064760 9859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:33:49.066088 9859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:33:49.075557 9865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:49.075560 9864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:49.075560 9867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:49.075832 9859 server_base.cc:1034] running on GCE node
I20260925 16:33:49.076202 9859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:33:49.076828 9859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:33:49.078014 9859 hybrid_clock.cc:648] HybridClock initialized: now 1790354029077975 us; error 52 us; skew 500 ppm
I20260925 16:33:49.079856 9859 webserver.cc:460] Webserver started at http://127.0.98.2:46309/ using document root <none> and password file <none>
I20260925 16:33:49.080426 9859 fs_manager.cc:359] Metadata directory not provided
I20260925 16:33:49.080515 9859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:33:49.080749 9859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:33:49.082845 9859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353966814071-392-0/minicluster-data/ts-1/data/instance:
uuid: "3784fe84b2474dd9982c1566c5f003c5"
format_stamp: "Formatted at 2026-09-25 16:33:49 on dist-test-slave-65pf"
server_key: "21316de085b54276edd2204e9f5a5c8b"
server_key_iv: "0efbc23adc52280b4f9c9b83af68c81a"
server_key_version: "encryptionkey@0"
I20260925 16:33:49.083400 9859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353966814071-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3784fe84b2474dd9982c1566c5f003c5"
format_stamp: "Formatted at 2026-09-25 16:33:49 on dist-test-slave-65pf"
server_key: "21316de085b54276edd2204e9f5a5c8b"
server_key_iv: "0efbc23adc52280b4f9c9b83af68c81a"
server_key_version: "encryptionkey@0"
I20260925 16:33:49.086951 9859 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 16:33:49.089433 9873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:49.090679 9859 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:33:49.090803 9859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353966814071-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353966814071-392-0/minicluster-data/ts-1/wal
uuid: "3784fe84b2474dd9982c1566c5f003c5"
format_stamp: "Formatted at 2026-09-25 16:33:49 on dist-test-slave-65pf"
server_key: "21316de085b54276edd2204e9f5a5c8b"
server_key_iv: "0efbc23adc52280b4f9c9b83af68c81a"
server_key_version: "encryptionkey@0"
I20260925 16:33:49.090920 9859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353966814071-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353966814071-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353966814071-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:33:49.106668 9859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:33:49.107386 9859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:33:49.107586 9859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:33:49.108179 9859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:33:49.109150 9859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:33:49.109220 9859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:49.109292 9859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:33:49.109344 9859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:49.118314 9859 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45595
I20260925 16:33:49.118335 9986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45595 every 8 connection(s)
I20260925 16:33:49.119311 9859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
I20260925 16:33:49.127632 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 9859
I20260925 16:33:49.127777 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353966814071-392-0/minicluster-data/ts-1/wal/instance
I20260925 16:33:49.128063 392 external_mini_cluster.cc:1428] Setting key 0b1b47caaf9f685cc7f80a64b57076a1
I20260925 16:33:49.129035 9707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:49.121120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46735 (local address 127.0.98.62:45361)
0925 16:33:49.121259 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:49.121262 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:49.122372 (+ 1110us) server_negotiation.cc:407] Connection header received
0925 16:33:49.123843 (+ 1471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:49.123846 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:49.123903 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:49.124016 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:49.124922 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:49.125448 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:49.126652 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:49.126871 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:49.126983 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:49.128279 (+ 1296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:49.128283 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:49.128286 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:33:49.128301 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:49.128324 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:33:49.128335 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:49.128424 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:49.128810 (+ 386us) server_negotiation.cc:299] Negotiation successful
0925 16:33:49.128887 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 16:33:49.129914 9989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:49.121535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45361 (local address 127.0.98.2:46735)
0925 16:33:49.122220 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:49.122261 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:49.123628 (+ 1367us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:49.124195 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:49.124202 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:49.124287 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:49.124782 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:49.124794 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:49.125579 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:49.125583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:49.126485 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:49.126495 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:49.127027 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:49.127032 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:49.127203 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:49.127942 (+ 739us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 16:33:49.127943 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:49.127982 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 16:33:49.128050 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:33:49.128430 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:33:49.128437 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:33:49.128441 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 16:33:49.128620 (+ 179us) client_negotiation.cc:240] Negotiation successful
0925 16:33:49.128890 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":119,"threads_started":1}
I20260925 16:33:49.130369 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353966814071-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353966814071-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353966814071-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45361
--builtin_ntp_servers=127.0.98.20:37315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 16:33:49.131117 9987 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45361
I20260925 16:33:49.131374 9987 heartbeater.cc:464] Registering TS with master...
I20260925 16:33:49.131997 9987 heartbeater.cc:511] Master 127.0.98.62:45361 requested a full tablet report, sending...
I20260925 16:33:49.133116 9668 ts_manager.cc:194] Registered new tserver with Master: 3784fe84b2474dd9982c1566c5f003c5 (127.0.98.2:45595)
I20260925 16:33:49.133687 9668 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:46735
W20260925 16:33:49.241520 9990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:33:49.241789 9990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:33:49.241827 9990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:33:49.245862 9990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:33:49.245950 9990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:33:49.246047 9990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 16:33:49.250764 9990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353966814071-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353966814071-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353966814071-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:33:49.251829 9990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:33:49.253206 9990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:33:49.262701 9996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:49.262715 9995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:49.262719 9998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:49.262761 9990 server_base.cc:1034] running on GCE node
I20260925 16:33:49.263253 9990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:33:49.263880 9990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:33:49.265065 9990 hybrid_clock.cc:648] HybridClock initialized: now 1790354029265052 us; error 26 us; skew 500 ppm
I20260925 16:33:49.267021 9990 webserver.cc:460] Webserver started at http://127.0.98.3:41199/ using document root <none> and password file <none>
I20260925 16:33:49.267607 9990 fs_manager.cc:359] Metadata directory not provided
I20260925 16:33:49.267704 9990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:33:49.267938 9990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:33:49.269950 9990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353966814071-392-0/minicluster-data/ts-2/data/instance:
uuid: "6a3084aa9dce4a378795fb77b2ecc091"
format_stamp: "Formatted at 2026-09-25 16:33:49 on dist-test-slave-65pf"
server_key: "0db92120ea2d0e6495459058d679aad5"
server_key_iv: "6678e3bfc8e257b19f7d4678864ab73b"
server_key_version: "encryptionkey@0"
I20260925 16:33:49.270521 9990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353966814071-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6a3084aa9dce4a378795fb77b2ecc091"
format_stamp: "Formatted at 2026-09-25 16:33:49 on dist-test-slave-65pf"
server_key: "0db92120ea2d0e6495459058d679aad5"
server_key_iv: "6678e3bfc8e257b19f7d4678864ab73b"
server_key_version: "encryptionkey@0"
I20260925 16:33:49.274103 9990 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 16:33:49.276589 10004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:49.277770 9990 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 16:33:49.277910 9990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353966814071-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353966814071-392-0/minicluster-data/ts-2/wal
uuid: "6a3084aa9dce4a378795fb77b2ecc091"
format_stamp: "Formatted at 2026-09-25 16:33:49 on dist-test-slave-65pf"
server_key: "0db92120ea2d0e6495459058d679aad5"
server_key_iv: "6678e3bfc8e257b19f7d4678864ab73b"
server_key_version: "encryptionkey@0"
I20260925 16:33:49.278021 9990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353966814071-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353966814071-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353966814071-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:33:49.293740 9990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:33:49.294423 9990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:33:49.294680 9990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:33:49.295279 9990 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:33:49.296208 9990 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:33:49.296283 9990 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:49.296351 9990 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:33:49.296403 9990 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:49.304862 9990 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33855
I20260925 16:33:49.304895 10117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33855 every 8 connection(s)
I20260925 16:33:49.305850 9990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
I20260925 16:33:49.307013 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 9990
I20260925 16:33:49.307123 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353966814071-392-0/minicluster-data/ts-2/wal/instance
I20260925 16:33:49.307458 392 external_mini_cluster.cc:1428] Setting key 27930b0ac007244ebf6fba72fc5380ff
I20260925 16:33:49.315281 9707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:49.307851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57751 (local address 127.0.98.62:45361)
0925 16:33:49.307989 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:49.307993 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:49.309215 (+ 1222us) server_negotiation.cc:407] Connection header received
0925 16:33:49.310798 (+ 1583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:49.310801 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:49.310845 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:49.310911 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:49.311711 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:49.312240 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:49.313551 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:49.313760 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:49.313840 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:49.314878 (+ 1038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:49.314881 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:49.314883 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:33:49.314894 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:49.314913 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:33:49.314920 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:49.314979 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:49.315120 (+ 141us) server_negotiation.cc:299] Negotiation successful
0925 16:33:49.315168 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 16:33:49.316092 10120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:49.308305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45361 (local address 127.0.98.3:57751)
0925 16:33:49.309079 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:49.309133 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:49.310552 (+ 1419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:49.311041 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:49.311047 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:49.311127 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:49.311598 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:49.311609 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:49.312348 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:49.312352 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:49.313448 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:49.313457 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:49.313896 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:49.313899 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:49.314039 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:49.314623 (+ 584us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 16:33:49.314625 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:49.314677 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 16:33:49.314739 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:33:49.314994 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:33:49.315000 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:33:49.315004 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 16:33:49.315100 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 16:33:49.315307 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":144,"threads_started":1}
I20260925 16:33:49.317214 10118 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45361
I20260925 16:33:49.317489 10118 heartbeater.cc:464] Registering TS with master...
I20260925 16:33:49.318156 10118 heartbeater.cc:511] Master 127.0.98.62:45361 requested a full tablet report, sending...
I20260925 16:33:49.319203 9668 ts_manager.cc:194] Registered new tserver with Master: 6a3084aa9dce4a378795fb77b2ecc091 (127.0.98.3:33855)
I20260925 16:33:49.319738 9668 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:57751
I20260925 16:33:49.321434 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 16:33:49.324096 9707 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:33:49.323290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52160 (local address 127.0.98.62:45361)
0925 16:33:49.323416 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:49.323421 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:49.323564 (+ 143us) server_negotiation.cc:407] Connection header received
0925 16:33:49.323645 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:49.323647 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:49.323683 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:49.323746 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:49.323982 (+ 236us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52160: BlockingRecv error: recv got EOF from 127.0.0.1:52160 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 16:33:49.324325 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 9728
I20260925 16:33:49.330101 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 9859
I20260925 16:33:49.335253 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 9990
I20260925 16:33:49.340269 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 9637
2026-09-25T16:33:49Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (848 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-25T16:33:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:33:49Z Disabled control of system clock
I20260925 16:33:49.368973 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353966814071-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353966814071-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353966814071-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32903
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32903
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 16:33:49.479439 10129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:33:49.479696 10129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:33:49.479732 10129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:33:49.483654 10129 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:33:49.483708 10129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:33:49.483724 10129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:33:49.483741 10129 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:33:49.483757 10129 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:33:49.488365 10129 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353966814071-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353966814071-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32903
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353966814071-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:33:49.489346 10129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:33:49.490685 10129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:33:49.499001 10137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:49.498986 10135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:49.499068 10134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:49.499230 10129 server_base.cc:1034] running on GCE node
I20260925 16:33:49.499855 10129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:33:49.500958 10129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:33:49.502163 10129 hybrid_clock.cc:648] HybridClock initialized: now 1790354029502147 us; error 28 us; skew 500 ppm
I20260925 16:33:49.504088 10129 webserver.cc:460] Webserver started at http://127.0.98.62:34317/ using document root <none> and password file <none>
I20260925 16:33:49.504666 10129 fs_manager.cc:359] Metadata directory not provided
I20260925 16:33:49.504758 10129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:33:49.504994 10129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:33:49.507117 10129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353966814071-392-0/minicluster-data/master-0/data/instance:
uuid: "58747d16ea3341d78a2437523d87702f"
format_stamp: "Formatted at 2026-09-25 16:33:49 on dist-test-slave-65pf"
server_key: "c68c36aa5540a0446c70312c2cacc563"
server_key_iv: "b0331d2848c9400c4d734928035f54aa"
server_key_version: "encryptionkey@0"
I20260925 16:33:49.507665 10129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353966814071-392-0/minicluster-data/master-0/wal/instance:
uuid: "58747d16ea3341d78a2437523d87702f"
format_stamp: "Formatted at 2026-09-25 16:33:49 on dist-test-slave-65pf"
server_key: "c68c36aa5540a0446c70312c2cacc563"
server_key_iv: "b0331d2848c9400c4d734928035f54aa"
server_key_version: "encryptionkey@0"
I20260925 16:33:49.511281 10129 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 16:33:49.513783 10143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:49.515067 10129 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:33:49.515206 10129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353966814071-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353966814071-392-0/minicluster-data/master-0/wal
uuid: "58747d16ea3341d78a2437523d87702f"
format_stamp: "Formatted at 2026-09-25 16:33:49 on dist-test-slave-65pf"
server_key: "c68c36aa5540a0446c70312c2cacc563"
server_key_iv: "b0331d2848c9400c4d734928035f54aa"
server_key_version: "encryptionkey@0"
I20260925 16:33:49.515314 10129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353966814071-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353966814071-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353966814071-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:33:49.519764 10129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:33:49.520349 10129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:33:49.520525 10129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:33:49.527040 10129 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32903
I20260925 16:33:49.527055 10195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32903 every 8 connection(s)
I20260925 16:33:49.527920 10129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:33:49.530941 10196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:33:49.534176 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 10129
I20260925 16:33:49.534291 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353966814071-392-0/minicluster-data/master-0/wal/instance
I20260925 16:33:49.534611 392 external_mini_cluster.cc:1428] Setting key eca61c807f6a8a6e465a1b060686ef49
I20260925 16:33:49.536671 10196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58747d16ea3341d78a2437523d87702f: Bootstrap starting.
I20260925 16:33:49.538960 10196 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58747d16ea3341d78a2437523d87702f: Neither blocks nor log segments found. Creating new log.
I20260925 16:33:49.539765 10196 log.cc:824] T 00000000000000000000000000000000 P 58747d16ea3341d78a2437523d87702f: Log is configured to *not* fsync() on all Append() calls
I20260925 16:33:49.540156 10199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:49.536052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34778 (local address 127.0.98.62:32903)
0925 16:33:49.536672 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:49.536686 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:49.536719 (+ 33us) server_negotiation.cc:407] Connection header received
0925 16:33:49.537380 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:49.537398 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:49.537731 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:49.538105 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:49.538920 (+ 815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:49.538924 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:49.538936 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:33:49.538946 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:49.538996 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:33:49.539012 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:49.539127 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:49.539251 (+ 124us) server_negotiation.cc:299] Negotiation successful
0925 16:33:49.539473 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":127,"threads_started":1}
I20260925 16:33:49.541754 10196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58747d16ea3341d78a2437523d87702f: No bootstrap required, opened a new log
I20260925 16:33:49.545094 10196 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58747d16ea3341d78a2437523d87702f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58747d16ea3341d78a2437523d87702f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32903 } }
I20260925 16:33:49.545295 10196 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58747d16ea3341d78a2437523d87702f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:33:49.545326 10196 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58747d16ea3341d78a2437523d87702f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58747d16ea3341d78a2437523d87702f, State: Initialized, Role: FOLLOWER
I20260925 16:33:49.545708 10196 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58747d16ea3341d78a2437523d87702f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58747d16ea3341d78a2437523d87702f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32903 } }
I20260925 16:33:49.545825 10196 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58747d16ea3341d78a2437523d87702f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:33:49.545861 10196 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58747d16ea3341d78a2437523d87702f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:33:49.545929 10196 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58747d16ea3341d78a2437523d87702f [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:33:49.546846 10196 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58747d16ea3341d78a2437523d87702f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58747d16ea3341d78a2437523d87702f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32903 } }
I20260925 16:33:49.547128 10196 leader_election.cc:302] T 00000000000000000000000000000000 P 58747d16ea3341d78a2437523d87702f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58747d16ea3341d78a2437523d87702f; no voters:
I20260925 16:33:49.547351 10196 leader_election.cc:288] T 00000000000000000000000000000000 P 58747d16ea3341d78a2437523d87702f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:33:49.547469 10201 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58747d16ea3341d78a2437523d87702f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:33:49.547678 10201 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58747d16ea3341d78a2437523d87702f [term 1 LEADER]: Becoming Leader. State: Replica: 58747d16ea3341d78a2437523d87702f, State: Running, Role: LEADER
I20260925 16:33:49.547991 10201 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58747d16ea3341d78a2437523d87702f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58747d16ea3341d78a2437523d87702f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32903 } }
I20260925 16:33:49.548529 10196 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58747d16ea3341d78a2437523d87702f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:33:49.549574 10202 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58747d16ea3341d78a2437523d87702f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58747d16ea3341d78a2437523d87702f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58747d16ea3341d78a2437523d87702f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32903 } } }
I20260925 16:33:49.549697 10202 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58747d16ea3341d78a2437523d87702f [sys.catalog]: This master's current role is: LEADER
I20260925 16:33:49.549957 10203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58747d16ea3341d78a2437523d87702f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58747d16ea3341d78a2437523d87702f. Latest consensus state: current_term: 1 leader_uuid: "58747d16ea3341d78a2437523d87702f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58747d16ea3341d78a2437523d87702f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32903 } } }
I20260925 16:33:49.550045 10203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58747d16ea3341d78a2437523d87702f [sys.catalog]: This master's current role is: LEADER
I20260925 16:33:49.550156 10210 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:33:49.552778 10210 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:33:49.558383 10210 catalog_manager.cc:1347] Generated new cluster ID: 0826c02988f54b1baa1ba043eb2a1654
I20260925 16:33:49.558454 10210 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:33:49.567988 10210 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:33:49.568295 10210 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:33:49.582116 10210 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58747d16ea3341d78a2437523d87702f: Generated new TSK 0
I20260925 16:33:49.582959 10210 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:33:49.588344 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353966814071-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353966814071-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353966814071-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32903
--builtin_ntp_servers=127.0.98.20:40619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 16:33:49.697621 10220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:33:49.697908 10220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:33:49.697989 10220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:33:49.702019 10220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:33:49.702106 10220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:33:49.702193 10220 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 16:33:49.706954 10220 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353966814071-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353966814071-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10220
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353966814071-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:33:49.707983 10220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:33:49.709344 10220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:33:49.718984 10228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:49.719000 10225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:49.718997 10226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:49.719174 10220 server_base.cc:1034] running on GCE node
I20260925 16:33:49.719677 10220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:33:49.720331 10220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:33:49.721530 10220 hybrid_clock.cc:648] HybridClock initialized: now 1790354029721523 us; error 31 us; skew 500 ppm
I20260925 16:33:49.723405 10220 webserver.cc:460] Webserver started at http://127.0.98.1:44829/ using document root <none> and password file <none>
I20260925 16:33:49.724046 10220 fs_manager.cc:359] Metadata directory not provided
I20260925 16:33:49.724146 10220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:33:49.724381 10220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:33:49.726423 10220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353966814071-392-0/minicluster-data/ts-0/data/instance:
uuid: "b380d6f4e63d4f27bb63b7e047d10253"
format_stamp: "Formatted at 2026-09-25 16:33:49 on dist-test-slave-65pf"
server_key: "9e3aad01f3aa8a346505cc86b7113d55"
server_key_iv: "51208f8264dd9dd080ab29b88565af80"
server_key_version: "encryptionkey@0"
I20260925 16:33:49.727029 10220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353966814071-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b380d6f4e63d4f27bb63b7e047d10253"
format_stamp: "Formatted at 2026-09-25 16:33:49 on dist-test-slave-65pf"
server_key: "9e3aad01f3aa8a346505cc86b7113d55"
server_key_iv: "51208f8264dd9dd080ab29b88565af80"
server_key_version: "encryptionkey@0"
I20260925 16:33:49.730525 10220 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 16:33:49.732859 10234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:49.734138 10220 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 16:33:49.734277 10220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353966814071-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353966814071-392-0/minicluster-data/ts-0/wal
uuid: "b380d6f4e63d4f27bb63b7e047d10253"
format_stamp: "Formatted at 2026-09-25 16:33:49 on dist-test-slave-65pf"
server_key: "9e3aad01f3aa8a346505cc86b7113d55"
server_key_iv: "51208f8264dd9dd080ab29b88565af80"
server_key_version: "encryptionkey@0"
I20260925 16:33:49.734378 10220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353966814071-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353966814071-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353966814071-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:33:49.738960 10220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:33:49.739521 10220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:33:49.739672 10220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:33:49.740161 10220 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:33:49.741037 10220 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:33:49.741088 10220 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:49.741130 10220 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:33:49.741156 10220 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:49.750914 10220 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45773
I20260925 16:33:49.750934 10347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45773 every 8 connection(s)
I20260925 16:33:49.751885 10220 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
I20260925 16:33:49.754710 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 10220
I20260925 16:33:49.754819 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353966814071-392-0/minicluster-data/ts-0/wal/instance
I20260925 16:33:49.755106 392 external_mini_cluster.cc:1428] Setting key b410872bd980a01e4f2fe6ac9d3b177f
I20260925 16:33:49.757040 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353966814071-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353966814071-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353966814071-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32903
--builtin_ntp_servers=127.0.98.20:40619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260925 16:33:49.758246 10199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:49.753683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35447 (local address 127.0.98.62:32903)
0925 16:33:49.753832 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:49.753835 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:49.754953 (+ 1118us) server_negotiation.cc:407] Connection header received
0925 16:33:49.756473 (+ 1520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:49.756476 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:49.756517 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:49.756593 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:49.757736 (+ 1143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:49.757740 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:49.757743 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:33:49.757759 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:49.757793 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:33:49.757802 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:49.757903 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:49.758057 (+ 154us) server_negotiation.cc:299] Negotiation successful
0925 16:33:49.758103 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 16:33:49.759337 10350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:49.754024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32903 (local address 127.0.98.1:35447)
0925 16:33:49.754795 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:49.754854 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:49.756260 (+ 1406us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:49.756764 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:49.756771 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:49.756864 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:49.757477 (+ 613us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 16:33:49.757479 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:49.757518 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 16:33:49.757585 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:33:49.757892 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:33:49.757901 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:33:49.757908 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 16:33:49.758100 (+ 192us) client_negotiation.cc:240] Negotiation successful
0925 16:33:49.758438 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":109,"threads_started":1}
I20260925 16:33:49.760571 10348 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32903
I20260925 16:33:49.760836 10348 heartbeater.cc:464] Registering TS with master...
I20260925 16:33:49.761240 10348 heartbeater.cc:511] Master 127.0.98.62:32903 requested a full tablet report, sending...
I20260925 16:33:49.762768 10160 ts_manager.cc:194] Registered new tserver with Master: b380d6f4e63d4f27bb63b7e047d10253 (127.0.98.1:45773)
W20260925 16:33:49.869360 10351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:33:49.869666 10351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:33:49.869747 10351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:33:49.873896 10351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:33:49.873972 10351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:33:49.874068 10351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 16:33:49.878901 10351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353966814071-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353966814071-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353966814071-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:33:49.879988 10351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:33:49.881433 10351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:33:49.890839 10357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:49.890890 10356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:49.890863 10359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:49.891206 10351 server_base.cc:1034] running on GCE node
I20260925 16:33:49.891620 10351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:33:49.892234 10351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:33:49.893417 10351 hybrid_clock.cc:648] HybridClock initialized: now 1790354029893390 us; error 52 us; skew 500 ppm
I20260925 16:33:49.895305 10351 webserver.cc:460] Webserver started at http://127.0.98.2:34121/ using document root <none> and password file <none>
I20260925 16:33:49.895889 10351 fs_manager.cc:359] Metadata directory not provided
I20260925 16:33:49.895990 10351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:33:49.896222 10351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:33:49.898240 10351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353966814071-392-0/minicluster-data/ts-1/data/instance:
uuid: "84db7c665e6649ba97bfec2f255e9a49"
format_stamp: "Formatted at 2026-09-25 16:33:49 on dist-test-slave-65pf"
server_key: "0e55bcb7c65f1526080cbb2c962d8034"
server_key_iv: "7ef50cb06ef60cf5b577904671f345d7"
server_key_version: "encryptionkey@0"
I20260925 16:33:49.898816 10351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353966814071-392-0/minicluster-data/ts-1/wal/instance:
uuid: "84db7c665e6649ba97bfec2f255e9a49"
format_stamp: "Formatted at 2026-09-25 16:33:49 on dist-test-slave-65pf"
server_key: "0e55bcb7c65f1526080cbb2c962d8034"
server_key_iv: "7ef50cb06ef60cf5b577904671f345d7"
server_key_version: "encryptionkey@0"
I20260925 16:33:49.902390 10351 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 16:33:49.904852 10365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:49.906062 10351 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:33:49.906194 10351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353966814071-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353966814071-392-0/minicluster-data/ts-1/wal
uuid: "84db7c665e6649ba97bfec2f255e9a49"
format_stamp: "Formatted at 2026-09-25 16:33:49 on dist-test-slave-65pf"
server_key: "0e55bcb7c65f1526080cbb2c962d8034"
server_key_iv: "7ef50cb06ef60cf5b577904671f345d7"
server_key_version: "encryptionkey@0"
I20260925 16:33:49.906306 10351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353966814071-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353966814071-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353966814071-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:33:49.910872 10351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:33:49.911456 10351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:33:49.911643 10351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:33:49.912166 10351 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:33:49.913041 10351 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:33:49.913110 10351 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:49.913173 10351 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:33:49.913208 10351 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:49.922081 10351 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37771
I20260925 16:33:49.922101 10478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37771 every 8 connection(s)
I20260925 16:33:49.923035 10351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
I20260925 16:33:49.923617 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 10351
I20260925 16:33:49.923740 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353966814071-392-0/minicluster-data/ts-1/wal/instance
I20260925 16:33:49.924063 392 external_mini_cluster.cc:1428] Setting key 247f969dec753f0c22269106bc07aa1e
I20260925 16:33:49.926270 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353966814071-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353966814071-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353966814071-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32903
--builtin_ntp_servers=127.0.98.20:40619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260925 16:33:49.930167 10199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:49.924991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59865 (local address 127.0.98.62:32903)
0925 16:33:49.925117 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:49.925120 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:49.926472 (+ 1352us) server_negotiation.cc:407] Connection header received
0925 16:33:49.928344 (+ 1872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:49.928347 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:49.928392 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:49.928493 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:49.929684 (+ 1191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:49.929687 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:49.929689 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:33:49.929701 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:49.929720 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:33:49.929728 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:49.929807 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:49.929989 (+ 182us) server_negotiation.cc:299] Negotiation successful
0925 16:33:49.930025 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 16:33:49.931387 10481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:49.925509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32903 (local address 127.0.98.2:59865)
0925 16:33:49.926319 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:49.926377 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:49.928129 (+ 1752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:49.928680 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:49.928692 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:49.928813 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:49.929380 (+ 567us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 16:33:49.929385 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:49.929444 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 16:33:49.929530 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:33:49.929825 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:33:49.929836 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:33:49.929845 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 16:33:49.930022 (+ 177us) client_negotiation.cc:240] Negotiation successful
0925 16:33:49.930275 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":131,"threads_started":1}
I20260925 16:33:49.932510 10479 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32903
I20260925 16:33:49.932765 10479 heartbeater.cc:464] Registering TS with master...
I20260925 16:33:49.933276 10479 heartbeater.cc:511] Master 127.0.98.62:32903 requested a full tablet report, sending...
I20260925 16:33:49.934372 10160 ts_manager.cc:194] Registered new tserver with Master: 84db7c665e6649ba97bfec2f255e9a49 (127.0.98.2:37771)
W20260925 16:33:50.038324 10482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:33:50.038648 10482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:33:50.038730 10482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:33:50.042830 10482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:33:50.042929 10482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:33:50.043030 10482 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 16:33:50.047741 10482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353966814071-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353966814071-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353966814071-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:33:50.048818 10482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:33:50.050670 10482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:33:50.059659 10487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:50.059701 10488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:50.059667 10490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:50.059927 10482 server_base.cc:1034] running on GCE node
I20260925 16:33:50.060427 10482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:33:50.061093 10482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:33:50.062352 10482 hybrid_clock.cc:648] HybridClock initialized: now 1790354030062334 us; error 30 us; skew 500 ppm
I20260925 16:33:50.064121 10482 webserver.cc:460] Webserver started at http://127.0.98.3:36475/ using document root <none> and password file <none>
I20260925 16:33:50.064677 10482 fs_manager.cc:359] Metadata directory not provided
I20260925 16:33:50.064765 10482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:33:50.064985 10482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:33:50.067036 10482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353966814071-392-0/minicluster-data/ts-2/data/instance:
uuid: "2c7c1e72850d48f0a4222543aa0f1d6b"
format_stamp: "Formatted at 2026-09-25 16:33:50 on dist-test-slave-65pf"
server_key: "7fcb4bfa03db3ea26f9c8b7d05ed1fc8"
server_key_iv: "f1897ef2ca9cb5ee67d9873c6901158c"
server_key_version: "encryptionkey@0"
I20260925 16:33:50.067569 10482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353966814071-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2c7c1e72850d48f0a4222543aa0f1d6b"
format_stamp: "Formatted at 2026-09-25 16:33:50 on dist-test-slave-65pf"
server_key: "7fcb4bfa03db3ea26f9c8b7d05ed1fc8"
server_key_iv: "f1897ef2ca9cb5ee67d9873c6901158c"
server_key_version: "encryptionkey@0"
I20260925 16:33:50.071036 10482 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 16:33:50.073338 10496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:50.074978 10482 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 16:33:50.075119 10482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353966814071-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353966814071-392-0/minicluster-data/ts-2/wal
uuid: "2c7c1e72850d48f0a4222543aa0f1d6b"
format_stamp: "Formatted at 2026-09-25 16:33:50 on dist-test-slave-65pf"
server_key: "7fcb4bfa03db3ea26f9c8b7d05ed1fc8"
server_key_iv: "f1897ef2ca9cb5ee67d9873c6901158c"
server_key_version: "encryptionkey@0"
I20260925 16:33:50.075227 10482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353966814071-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353966814071-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353966814071-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:33:50.079617 10482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:33:50.080188 10482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:33:50.080367 10482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:33:50.080896 10482 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:33:50.081827 10482 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:33:50.081895 10482 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:50.081964 10482 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:33:50.082015 10482 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:50.090646 10482 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44073
I20260925 16:33:50.090667 10609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44073 every 8 connection(s)
I20260925 16:33:50.091503 10482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
I20260925 16:33:50.092828 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 10482
I20260925 16:33:50.092938 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353966814071-392-0/minicluster-data/ts-2/wal/instance
I20260925 16:33:50.093261 392 external_mini_cluster.cc:1428] Setting key 55e161d029f1148845b6a1572fc735e2
I20260925 16:33:50.097440 10199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:50.093292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36967 (local address 127.0.98.62:32903)
0925 16:33:50.093441 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:50.093445 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:50.094673 (+ 1228us) server_negotiation.cc:407] Connection header received
0925 16:33:50.096033 (+ 1360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:50.096035 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:50.096070 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:50.096143 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:50.097091 (+ 948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:50.097094 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:50.097096 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:33:50.097106 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:50.097125 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:33:50.097133 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:50.097206 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:50.097302 (+ 96us) server_negotiation.cc:299] Negotiation successful
0925 16:33:50.097338 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 16:33:50.098299 10612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:50.093761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32903 (local address 127.0.98.3:36967)
0925 16:33:50.094507 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:50.094563 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:50.095847 (+ 1284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:50.096267 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:50.096273 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:50.096356 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:50.096855 (+ 499us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 16:33:50.096857 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:50.096894 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 16:33:50.096957 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:33:50.097198 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:33:50.097201 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:33:50.097205 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 16:33:50.097317 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 16:33:50.097524 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":114,"threads_started":1}
I20260925 16:33:50.099381 10610 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32903
I20260925 16:33:50.099612 10610 heartbeater.cc:464] Registering TS with master...
I20260925 16:33:50.100001 10610 heartbeater.cc:511] Master 127.0.98.62:32903 requested a full tablet report, sending...
I20260925 16:33:50.100926 10160 ts_manager.cc:194] Registered new tserver with Master: 2c7c1e72850d48f0a4222543aa0f1d6b (127.0.98.3:44073)
I20260925 16:33:50.107034 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 16:33:50.109740 10199 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:33:50.108923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34790 (local address 127.0.98.62:32903)
0925 16:33:50.109026 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:50.109029 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:50.109214 (+ 185us) server_negotiation.cc:407] Connection header received
0925 16:33:50.109305 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:50.109307 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:50.109336 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:50.109410 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:50.109593 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34790: BlockingRecv error: recv got EOF from 127.0.0.1:34790 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 16:33:50.109958 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 10220
I20260925 16:33:50.115183 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 10351
I20260925 16:33:50.120411 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 10482
I20260925 16:33:50.125525 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 10129
2026-09-25T16:33:50Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (785 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:33:50 dist-test-slave-65pf krb5kdc[10620](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:33:50 dist-test-slave-65pf krb5kdc[10620](info): set up 2 sockets
Sep 25 16:33:50 dist-test-slave-65pf krb5kdc[10620](info): commencing operation
krb5kdc: starting...
W20260925 16:33:51.163678 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/krb5kdc/test-user.keytab.
Sep 25 16:33:51 dist-test-slave-65pf krb5kdc[10620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:33:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:33:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 16:33:51.287369 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41653
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41653
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:33:51.393733 10633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:33:51.394018 10633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:33:51.394102 10633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:33:51.397925 10633 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:33:51.398020 10633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:33:51.398056 10633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:33:51.398092 10633 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:33:51.398121 10633 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:33:51.403280 10633 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41653
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:33:51.404318 10633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:33:51.405308 10633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:33:51.413545 10639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:51.413487 10638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:51.413509 10641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:51.413689 10633 server_base.cc:1034] running on GCE node
I20260925 16:33:51.414314 10633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:33:51.415416 10633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:33:51.416610 10633 hybrid_clock.cc:648] HybridClock initialized: now 1790354031416602 us; error 29 us; skew 500 ppm
Sep 25 16:33:51 dist-test-slave-65pf krb5kdc[10620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:33:51.427165 10633 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 16:33:51.428092 10633 webserver.cc:460] Webserver started at http://127.0.98.62:36751/ using document root <none> and password file <none>
I20260925 16:33:51.428639 10633 fs_manager.cc:359] Metadata directory not provided
I20260925 16:33:51.428726 10633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:33:51.428951 10633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:33:51.431005 10633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/minicluster-data/master-0/data/instance:
uuid: "70685a793b014ad3a627bbf5e00e4ab0"
format_stamp: "Formatted at 2026-09-25 16:33:51 on dist-test-slave-65pf"
server_key: "fc5f95243dc04e939da864db8efec0c3"
server_key_iv: "80985d20dac3883a1bca98180a89fd87"
server_key_version: "encryptionkey@0"
I20260925 16:33:51.431555 10633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/minicluster-data/master-0/wal/instance:
uuid: "70685a793b014ad3a627bbf5e00e4ab0"
format_stamp: "Formatted at 2026-09-25 16:33:51 on dist-test-slave-65pf"
server_key: "fc5f95243dc04e939da864db8efec0c3"
server_key_iv: "80985d20dac3883a1bca98180a89fd87"
server_key_version: "encryptionkey@0"
I20260925 16:33:51.435037 10633 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 16:33:51.437292 10648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:51.438397 10633 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:33:51.438544 10633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/minicluster-data/master-0/wal
uuid: "70685a793b014ad3a627bbf5e00e4ab0"
format_stamp: "Formatted at 2026-09-25 16:33:51 on dist-test-slave-65pf"
server_key: "fc5f95243dc04e939da864db8efec0c3"
server_key_iv: "80985d20dac3883a1bca98180a89fd87"
server_key_version: "encryptionkey@0"
I20260925 16:33:51.438704 10633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:33:51.470080 10633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:33:51.473168 10633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:33:51.473383 10633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:33:51.480436 10633 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41653
I20260925 16:33:51.480716 10700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41653 every 8 connection(s)
I20260925 16:33:51.481482 10633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:33:51.483137 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 10633
I20260925 16:33:51.483227 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/minicluster-data/master-0/wal/instance
I20260925 16:33:51.483515 392 external_mini_cluster.cc:1428] Setting key d675bf0e17ea64b9b7824ef1a4d4eae9
I20260925 16:33:51.484447 10701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:33:51.490994 10701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70685a793b014ad3a627bbf5e00e4ab0: Bootstrap starting.
Sep 25 16:33:51 dist-test-slave-65pf krb5kdc[10620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:33:51.493389 10701 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70685a793b014ad3a627bbf5e00e4ab0: Neither blocks nor log segments found. Creating new log.
I20260925 16:33:51.494282 10701 log.cc:824] T 00000000000000000000000000000000 P 70685a793b014ad3a627bbf5e00e4ab0: Log is configured to *not* fsync() on all Append() calls
I20260925 16:33:51.496376 10701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70685a793b014ad3a627bbf5e00e4ab0: No bootstrap required, opened a new log
I20260925 16:33:51.498046 10704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:51.484911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39088 (local address 127.0.98.62:41653)
0925 16:33:51.485316 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:51.485335 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:51.485376 (+ 41us) server_negotiation.cc:407] Connection header received
0925 16:33:51.486228 (+ 852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:51.486250 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:51.486697 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:51.487059 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:51.487698 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:51.489167 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:51.489979 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:51.490708 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:51.490859 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:51.493442 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:51.493448 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:51.493462 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:51.493491 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:51.495296 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:51.495787 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:51.495803 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:51.495807 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:51.495909 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:51.496198 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:51.496201 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:51.496202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:51.496630 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:51.496763 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:51.497068 (+ 305us) server_negotiation.cc:299] Negotiation successful
0925 16:33:51.497358 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":110,"threads_started":1}
I20260925 16:33:51.500329 10701 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70685a793b014ad3a627bbf5e00e4ab0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70685a793b014ad3a627bbf5e00e4ab0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41653 } }
I20260925 16:33:51.500559 10701 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70685a793b014ad3a627bbf5e00e4ab0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:33:51.500635 10701 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70685a793b014ad3a627bbf5e00e4ab0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70685a793b014ad3a627bbf5e00e4ab0, State: Initialized, Role: FOLLOWER
I20260925 16:33:51.501037 10701 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70685a793b014ad3a627bbf5e00e4ab0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70685a793b014ad3a627bbf5e00e4ab0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41653 } }
I20260925 16:33:51.501179 10701 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70685a793b014ad3a627bbf5e00e4ab0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:33:51.501261 10701 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70685a793b014ad3a627bbf5e00e4ab0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:33:51.501379 10701 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70685a793b014ad3a627bbf5e00e4ab0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:33:51.502398 10701 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70685a793b014ad3a627bbf5e00e4ab0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70685a793b014ad3a627bbf5e00e4ab0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41653 } }
I20260925 16:33:51.502786 10701 leader_election.cc:302] T 00000000000000000000000000000000 P 70685a793b014ad3a627bbf5e00e4ab0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70685a793b014ad3a627bbf5e00e4ab0; no voters:
I20260925 16:33:51.503111 10701 leader_election.cc:288] T 00000000000000000000000000000000 P 70685a793b014ad3a627bbf5e00e4ab0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:33:51.503185 10706 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70685a793b014ad3a627bbf5e00e4ab0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:33:51.503459 10706 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70685a793b014ad3a627bbf5e00e4ab0 [term 1 LEADER]: Becoming Leader. State: Replica: 70685a793b014ad3a627bbf5e00e4ab0, State: Running, Role: LEADER
I20260925 16:33:51.503782 10706 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70685a793b014ad3a627bbf5e00e4ab0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70685a793b014ad3a627bbf5e00e4ab0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41653 } }
I20260925 16:33:51.504328 10701 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70685a793b014ad3a627bbf5e00e4ab0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:33:51.505553 10708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70685a793b014ad3a627bbf5e00e4ab0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70685a793b014ad3a627bbf5e00e4ab0. Latest consensus state: current_term: 1 leader_uuid: "70685a793b014ad3a627bbf5e00e4ab0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70685a793b014ad3a627bbf5e00e4ab0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41653 } } }
I20260925 16:33:51.505565 10707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70685a793b014ad3a627bbf5e00e4ab0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70685a793b014ad3a627bbf5e00e4ab0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70685a793b014ad3a627bbf5e00e4ab0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41653 } } }
I20260925 16:33:51.505682 10708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70685a793b014ad3a627bbf5e00e4ab0 [sys.catalog]: This master's current role is: LEADER
I20260925 16:33:51.505686 10707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70685a793b014ad3a627bbf5e00e4ab0 [sys.catalog]: This master's current role is: LEADER
I20260925 16:33:51.506055 10715 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:33:51.509176 10715 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:33:51.514983 10715 catalog_manager.cc:1347] Generated new cluster ID: f53fd6a417b9426789ef944ea4b9752c
I20260925 16:33:51.515061 10715 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:33:51.523092 10715 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:33:51.524005 10715 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:33:51.529595 10715 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70685a793b014ad3a627bbf5e00e4ab0: Generated new TSK 0
I20260925 16:33:51.530251 10715 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 16:33:51.559522 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41653
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:33:51.669329 10727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:33:51.669569 10727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:33:51.669641 10727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:33:51.673645 10727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:33:51.673702 10727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:33:51.673790 10727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 16:33:51.678952 10727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10727
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:33:51.679943 10727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:33:51.680944 10727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:33:51.690963 10732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:51.690980 10733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:51.690989 10735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:51.691205 10727 server_base.cc:1034] running on GCE node
I20260925 16:33:51.691568 10727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:33:51.692169 10727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:33:51.693357 10727 hybrid_clock.cc:648] HybridClock initialized: now 1790354031693338 us; error 31 us; skew 500 ppm
Sep 25 16:33:51 dist-test-slave-65pf krb5kdc[10620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:33:51.696111 10727 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 16:33:51.696898 10727 webserver.cc:460] Webserver started at http://127.0.98.1:38977/ using document root <none> and password file <none>
I20260925 16:33:51.697434 10727 fs_manager.cc:359] Metadata directory not provided
I20260925 16:33:51.697494 10727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:33:51.697652 10727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:33:51.699667 10727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/minicluster-data/ts-0/data/instance:
uuid: "0069902798724e2d8482c58cfed66f1c"
format_stamp: "Formatted at 2026-09-25 16:33:51 on dist-test-slave-65pf"
server_key: "10a0ad62038cd682f0ccead3ef27613d"
server_key_iv: "226fde936a8347f364124a6a9b5eeac2"
server_key_version: "encryptionkey@0"
I20260925 16:33:51.700165 10727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0069902798724e2d8482c58cfed66f1c"
format_stamp: "Formatted at 2026-09-25 16:33:51 on dist-test-slave-65pf"
server_key: "10a0ad62038cd682f0ccead3ef27613d"
server_key_iv: "226fde936a8347f364124a6a9b5eeac2"
server_key_version: "encryptionkey@0"
I20260925 16:33:51.703732 10727 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260925 16:33:51.706290 10742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:51.707660 10727 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 16:33:51.707775 10727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/minicluster-data/ts-0/wal
uuid: "0069902798724e2d8482c58cfed66f1c"
format_stamp: "Formatted at 2026-09-25 16:33:51 on dist-test-slave-65pf"
server_key: "10a0ad62038cd682f0ccead3ef27613d"
server_key_iv: "226fde936a8347f364124a6a9b5eeac2"
server_key_version: "encryptionkey@0"
I20260925 16:33:51.707863 10727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:33:51.727144 10727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:33:51.729930 10727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:33:51.730084 10727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:33:51.730652 10727 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:33:51.731415 10727 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:33:51.731459 10727 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:51.731498 10727 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:33:51.731513 10727 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:51.740073 10727 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46429
I20260925 16:33:51.740144 10855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46429 every 8 connection(s)
I20260925 16:33:51.741011 10727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
I20260925 16:33:51.745615 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 10727
I20260925 16:33:51.745736 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/minicluster-data/ts-0/wal/instance
I20260925 16:33:51.746026 392 external_mini_cluster.cc:1428] Setting key 3a8a874829a6fca8dae6c0f9c50d4b17
Sep 25 16:33:51 dist-test-slave-65pf krb5kdc[10620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:33:51.755651 10704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:51.742950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36565 (local address 127.0.98.62:41653)
0925 16:33:51.743087 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:51.743090 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:51.744118 (+ 1028us) server_negotiation.cc:407] Connection header received
0925 16:33:51.744956 (+ 838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:51.744959 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:51.745003 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:51.745075 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:51.746491 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:51.747157 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:51.748365 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:51.748786 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:51.748913 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:51.751808 (+ 2895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:51.751814 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:51.751817 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:51.751846 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:51.753523 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:51.754003 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:51.754006 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:51.754008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:51.754109 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:51.754431 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:51.754437 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:51.754439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:51.754708 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:51.754806 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:51.755305 (+ 499us) server_negotiation.cc:299] Negotiation successful
0925 16:33:51.755453 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 16:33:51.756414 10858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:51.743343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41653 (local address 127.0.98.1:36565)
0925 16:33:51.743981 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:51.744020 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:51.744776 (+ 756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:51.745235 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:51.745241 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:51.745703 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:51.746299 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:51.746314 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:51.747287 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:51.747291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:51.748233 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:51.748242 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:51.748959 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:51.748965 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:51.749189 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:51.749848 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:51.749873 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:51.751639 (+ 1766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:33:51.753634 (+ 1995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:51.753639 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:51.753651 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:51.753894 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:51.754211 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:51.754213 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:51.754215 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:51.754337 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:51.754817 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:33:51.754823 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:33:51.755054 (+ 231us) client_negotiation.cc:769] Sending connection context
0925 16:33:51.755213 (+ 159us) client_negotiation.cc:240] Negotiation successful
0925 16:33:51.755464 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":108,"threads_started":1}
I20260925 16:33:51.757655 10856 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41653
I20260925 16:33:51.757894 10856 heartbeater.cc:464] Registering TS with master...
I20260925 16:33:51.758373 10856 heartbeater.cc:511] Master 127.0.98.62:41653 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260925 16:33:51.759912 10665 ts_manager.cc:194] Registered new tserver with Master: 0069902798724e2d8482c58cfed66f1c (127.0.98.1:46429)
I20260925 16:33:51.761022 10665 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36565
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 16:33:51.774082 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41653
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:33:51.882599 10861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:33:51.882925 10861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:33:51.883006 10861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:33:51.887269 10861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:33:51.887362 10861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:33:51.887455 10861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 16:33:51.892793 10861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10861
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:33:51.893875 10861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:33:51.894979 10861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:33:51.905023 10866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:51.905032 10869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:51.905025 10867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:51.905329 10861 server_base.cc:1034] running on GCE node
I20260925 16:33:51.905723 10861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:33:51.906453 10861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:33:51.907646 10861 hybrid_clock.cc:648] HybridClock initialized: now 1790354031907634 us; error 35 us; skew 500 ppm
Sep 25 16:33:51 dist-test-slave-65pf krb5kdc[10620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:33:51.910233 10861 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 16:33:51.911187 10861 webserver.cc:460] Webserver started at http://127.0.98.2:39947/ using document root <none> and password file <none>
I20260925 16:33:51.911811 10861 fs_manager.cc:359] Metadata directory not provided
I20260925 16:33:51.911903 10861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:33:51.912143 10861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:33:51.914156 10861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/minicluster-data/ts-1/data/instance:
uuid: "a091ffa1fa3e4a2787ceca008d229dde"
format_stamp: "Formatted at 2026-09-25 16:33:51 on dist-test-slave-65pf"
server_key: "60ad1a3fda6701172367f514776a93e4"
server_key_iv: "b2cc484058f15212fcd20fac9e13c048"
server_key_version: "encryptionkey@0"
I20260925 16:33:51.914817 10861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a091ffa1fa3e4a2787ceca008d229dde"
format_stamp: "Formatted at 2026-09-25 16:33:51 on dist-test-slave-65pf"
server_key: "60ad1a3fda6701172367f514776a93e4"
server_key_iv: "b2cc484058f15212fcd20fac9e13c048"
server_key_version: "encryptionkey@0"
I20260925 16:33:51.918468 10861 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 16:33:51.920941 10876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:51.922142 10861 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:33:51.922273 10861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/minicluster-data/ts-1/wal
uuid: "a091ffa1fa3e4a2787ceca008d229dde"
format_stamp: "Formatted at 2026-09-25 16:33:51 on dist-test-slave-65pf"
server_key: "60ad1a3fda6701172367f514776a93e4"
server_key_iv: "b2cc484058f15212fcd20fac9e13c048"
server_key_version: "encryptionkey@0"
I20260925 16:33:51.922386 10861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:33:51.947052 10861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:33:51.951682 10861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:33:51.951887 10861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:33:51.952502 10861 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:33:51.953320 10861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:33:51.953366 10861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:51.953440 10861 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:33:51.953472 10861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:51.961781 10861 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43229
I20260925 16:33:51.961853 10989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43229 every 8 connection(s)
I20260925 16:33:51.962777 10861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
I20260925 16:33:51.970279 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 10861
I20260925 16:33:51.970392 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/minicluster-data/ts-1/wal/instance
I20260925 16:33:51.970876 392 external_mini_cluster.cc:1428] Setting key 4a873015f04d2b3d094ddf3e5d40b9ce
Sep 25 16:33:51 dist-test-slave-65pf krb5kdc[10620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:33:51.977082 10704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:51.964555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37607 (local address 127.0.98.62:41653)
0925 16:33:51.964714 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:51.964719 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:51.965728 (+ 1009us) server_negotiation.cc:407] Connection header received
0925 16:33:51.966730 (+ 1002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:51.966734 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:51.966785 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:51.966880 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:51.968022 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:51.968568 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:51.969661 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:51.969896 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:51.970083 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:51.972941 (+ 2858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:51.972946 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:51.972954 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:51.972976 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:51.974847 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:51.975345 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:51.975351 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:51.975354 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:51.975443 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:51.975838 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:51.975844 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:51.975849 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:51.976135 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:51.976241 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:51.976735 (+ 494us) server_negotiation.cc:299] Negotiation successful
0925 16:33:51.976883 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 16:33:51.977805 10992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:51.964951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41653 (local address 127.0.98.2:37607)
0925 16:33:51.965578 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:51.965619 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:51.966391 (+ 772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:51.967043 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:51.967049 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:51.967396 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:51.967869 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:51.967882 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:51.968677 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:51.968679 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:51.969545 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:51.969553 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:51.970117 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:51.970120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:51.970265 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:51.971046 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:51.971062 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:51.972777 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:33:51.974977 (+ 2200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:51.974982 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:51.974994 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:51.975243 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:51.975543 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:51.975546 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:51.975548 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:51.975729 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:51.976238 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:33:51.976243 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:33:51.976482 (+ 239us) client_negotiation.cc:769] Sending connection context
0925 16:33:51.976645 (+ 163us) client_negotiation.cc:240] Negotiation successful
0925 16:33:51.976899 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":111,"threads_started":1}
I20260925 16:33:51.979049 10990 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41653
I20260925 16:33:51.979311 10990 heartbeater.cc:464] Registering TS with master...
I20260925 16:33:51.979911 10990 heartbeater.cc:511] Master 127.0.98.62:41653 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260925 16:33:51.981079 10665 ts_manager.cc:194] Registered new tserver with Master: a091ffa1fa3e4a2787ceca008d229dde (127.0.98.2:43229)
I20260925 16:33:51.981624 10665 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37607
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 16:33:51.999109 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41653
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:33:52.109334 10995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:33:52.109612 10995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:33:52.109656 10995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:33:52.113890 10995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:33:52.113957 10995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:33:52.114048 10995 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 16:33:52.119480 10995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:33:52.120651 10995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:33:52.121701 10995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:33:52.131434 11001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:52.131464 11003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:33:52.131428 11000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:33:52.131727 10995 server_base.cc:1034] running on GCE node
I20260925 16:33:52.132154 10995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:33:52.132766 10995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:33:52.133955 10995 hybrid_clock.cc:648] HybridClock initialized: now 1790354032133941 us; error 37 us; skew 500 ppm
Sep 25 16:33:52 dist-test-slave-65pf krb5kdc[10620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354032, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:33:52.136755 10995 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 16:33:52.137586 10995 webserver.cc:460] Webserver started at http://127.0.98.3:46035/ using document root <none> and password file <none>
I20260925 16:33:52.138127 10995 fs_manager.cc:359] Metadata directory not provided
I20260925 16:33:52.138217 10995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:33:52.138458 10995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:33:52.140453 10995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/minicluster-data/ts-2/data/instance:
uuid: "8ac8324364404f14a4f3114b0cd21ebd"
format_stamp: "Formatted at 2026-09-25 16:33:52 on dist-test-slave-65pf"
server_key: "43fea112488177c109667ba45edde7ef"
server_key_iv: "9251b349bf6ff359ec70091a4f5e3bb3"
server_key_version: "encryptionkey@0"
I20260925 16:33:52.141001 10995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8ac8324364404f14a4f3114b0cd21ebd"
format_stamp: "Formatted at 2026-09-25 16:33:52 on dist-test-slave-65pf"
server_key: "43fea112488177c109667ba45edde7ef"
server_key_iv: "9251b349bf6ff359ec70091a4f5e3bb3"
server_key_version: "encryptionkey@0"
I20260925 16:33:52.144701 10995 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 16:33:52.147086 11010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:52.148239 10995 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:33:52.148382 10995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/minicluster-data/ts-2/wal
uuid: "8ac8324364404f14a4f3114b0cd21ebd"
format_stamp: "Formatted at 2026-09-25 16:33:52 on dist-test-slave-65pf"
server_key: "43fea112488177c109667ba45edde7ef"
server_key_iv: "9251b349bf6ff359ec70091a4f5e3bb3"
server_key_version: "encryptionkey@0"
I20260925 16:33:52.148494 10995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:33:52.168334 10995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:33:52.171607 10995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:33:52.171811 10995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:33:52.172395 10995 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:33:52.173184 10995 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:33:52.173255 10995 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:52.173322 10995 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:33:52.173375 10995 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:33:52.182319 10995 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36693
I20260925 16:33:52.182340 11123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36693 every 8 connection(s)
I20260925 16:33:52.183189 10995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
I20260925 16:33:52.185441 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 10995
I20260925 16:33:52.185535 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/minicluster-data/ts-2/wal/instance
I20260925 16:33:52.185846 392 external_mini_cluster.cc:1428] Setting key 69d48b3862ab5deb234c518e74f7cdc5
Sep 25 16:33:52 dist-test-slave-65pf krb5kdc[10620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354032, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:33:52.195869 10704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:52.184994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59515 (local address 127.0.98.62:41653)
0925 16:33:52.185155 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:52.185159 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:52.186196 (+ 1037us) server_negotiation.cc:407] Connection header received
0925 16:33:52.187222 (+ 1026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:52.187225 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:52.187276 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:52.187369 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:52.188630 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:52.189154 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:52.190270 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:52.190496 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:52.190629 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:52.192875 (+ 2246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:52.192878 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:52.192880 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:52.192901 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:52.194139 (+ 1238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:52.194647 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:52.194650 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:52.194651 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:52.194720 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:52.195016 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:52.195018 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:52.195019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:52.195171 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:52.195242 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:52.195663 (+ 421us) server_negotiation.cc:299] Negotiation successful
0925 16:33:52.195753 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 16:33:52.196636 11126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:52.185321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41653 (local address 127.0.98.3:59515)
0925 16:33:52.186049 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:52.186100 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:52.187021 (+ 921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:52.187560 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:52.187567 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:52.188022 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:33:52.188493 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:52.188506 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:52.189280 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:52.189289 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:52.190146 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:52.190155 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:52.190660 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:52.190664 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:52.190839 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:52.191363 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:33:52.191377 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:33:52.192731 (+ 1354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:33:52.194262 (+ 1531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:52.194266 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:52.194277 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:52.194515 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:52.194800 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:33:52.194802 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:33:52.194804 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:33:52.194938 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:33:52.195253 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:33:52.195258 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:33:52.195472 (+ 214us) client_negotiation.cc:769] Sending connection context
0925 16:33:52.195619 (+ 147us) client_negotiation.cc:240] Negotiation successful
0925 16:33:52.195848 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":138,"threads_started":1}
I20260925 16:33:52.197719 11124 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41653
I20260925 16:33:52.197953 11124 heartbeater.cc:464] Registering TS with master...
I20260925 16:33:52.198479 11124 heartbeater.cc:511] Master 127.0.98.62:41653 requested a full tablet report, sending...
I20260925 16:33:52.199594 10665 ts_manager.cc:194] Registered new tserver with Master: 8ac8324364404f14a4f3114b0cd21ebd (127.0.98.3:36693)
I20260925 16:33:52.200145 10665 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59515
I20260925 16:33:52.211117 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 16:33:52.219677 10704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:52.213004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39096 (local address 127.0.98.62:41653)
0925 16:33:52.213130 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:52.213135 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:52.213320 (+ 185us) server_negotiation.cc:407] Connection header received
0925 16:33:52.213482 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:52.213485 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:52.213535 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:52.213608 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:33:52.214263 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:52.214847 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:52.215663 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:52.215861 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:52.215934 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:52.217120 (+ 1186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:33:52.217123 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:33:52.217125 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:33:52.217145 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:33:52.218361 (+ 1216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:52.218736 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:52.218738 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:52.218740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:52.218811 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:33:52.219034 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:33:52.219036 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:33:52.219037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:33:52.219226 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:33:52.219326 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:52.219478 (+ 152us) server_negotiation.cc:299] Negotiation successful
0925 16:33:52.219564 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 16:33:52.222318 10665 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39096:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 16:33:52.225270 10665 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:33:52.239499 11139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:52.234038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35574 (local address 127.0.98.3:36693)
0925 16:33:52.234347 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:52.234358 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:52.234437 (+ 79us) server_negotiation.cc:407] Connection header received
0925 16:33:52.234690 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:52.234697 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:52.234906 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:52.235049 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:33:52.235471 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:52.236653 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:52.238203 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:52.238813 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:52.238934 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:52.239084 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:52.239247 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 16:33:52.239344 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":136,"threads_started":1}
I20260925 16:33:52.239493 11138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:52.233350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46429 (local address 127.0.0.1:50158)
0925 16:33:52.234205 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:52.234251 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:52.234457 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:52.235126 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:52.235131 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:52.235150 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:33:52.235244 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:52.235249 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:52.236783 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:52.236786 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:52.237932 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:52.237940 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:52.239016 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:52.239020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:52.239102 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:52.239122 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 16:33:52.239181 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 16:33:52.239318 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":703,"mutex_wait_us":56,"thread_start_us":95,"threads_started":1}
I20260925 16:33:52.239787 11140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:52.234037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43936 (local address 127.0.98.2:43229)
0925 16:33:52.235062 (+ 1025us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:52.235074 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:52.235097 (+ 23us) server_negotiation.cc:407] Connection header received
0925 16:33:52.235164 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:52.235169 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:52.235335 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:52.235663 (+ 328us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:33:52.235779 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:52.236484 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:52.238142 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:52.238917 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:52.239011 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:52.239146 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:52.239556 (+ 410us) server_negotiation.cc:299] Negotiation successful
0925 16:33:52.239650 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":926,"thread_start_us":124,"threads_started":1}
I20260925 16:33:52.239966 11137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:52.233519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50158 (local address 127.0.98.1:46429)
0925 16:33:52.233893 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:52.233921 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:52.234343 (+ 422us) server_negotiation.cc:407] Connection header received
0925 16:33:52.234670 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:52.234677 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:52.234893 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:52.235343 (+ 450us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:33:52.235508 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:52.236662 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:52.238051 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:52.238875 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:52.239560 (+ 685us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:52.239685 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:52.239728 (+ 43us) server_negotiation.cc:299] Negotiation successful
0925 16:33:52.239817 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":107,"threads_started":1}
I20260925 16:33:52.240015 11136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:52.233577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43229 (local address 127.0.0.1:43936)
0925 16:33:52.234732 (+ 1155us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:52.234744 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:52.234814 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:52.235478 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:52.235487 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:52.235501 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:33:52.235557 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:52.235561 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:52.236614 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:52.236620 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:52.238035 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:52.238045 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:52.239201 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:52.239205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:52.239281 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:52.239300 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 16:33:52.239374 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 16:33:52.239437 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1099,"mutex_wait_us":7,"thread_start_us":92,"threads_started":1}
I20260925 16:33:52.239490 11135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:52.233237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36693 (local address 127.0.0.1:35574)
0925 16:33:52.234340 (+ 1103us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:52.234354 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:52.234457 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:52.235219 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:52.235222 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:52.235239 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:33:52.235336 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:52.235344 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:52.236943 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:52.236946 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:52.238039 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:52.238045 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:52.238938 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:52.238941 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:52.239051 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:52.239086 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 16:33:52.239219 (+ 133us) client_negotiation.cc:240] Negotiation successful
0925 16:33:52.239344 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"mutex_wait_us":45,"thread_start_us":95,"threads_started":1}
I20260925 16:33:52.243008 10790 tablet_service.cc:1467] Processing CreateTablet for tablet 63d0cf6f83ea46dbb50c03192a5da0d2 (DEFAULT_TABLE table=test-table [id=d0f2c02592c940498f4dd8344f04b17d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:33:52.243013 11058 tablet_service.cc:1467] Processing CreateTablet for tablet 63d0cf6f83ea46dbb50c03192a5da0d2 (DEFAULT_TABLE table=test-table [id=d0f2c02592c940498f4dd8344f04b17d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:33:52.243012 10924 tablet_service.cc:1467] Processing CreateTablet for tablet 63d0cf6f83ea46dbb50c03192a5da0d2 (DEFAULT_TABLE table=test-table [id=d0f2c02592c940498f4dd8344f04b17d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:33:52.243970 10790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63d0cf6f83ea46dbb50c03192a5da0d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:33:52.244001 11058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63d0cf6f83ea46dbb50c03192a5da0d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:33:52.243997 10924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63d0cf6f83ea46dbb50c03192a5da0d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:33:52.250011 11141 tablet_bootstrap.cc:492] T 63d0cf6f83ea46dbb50c03192a5da0d2 P 0069902798724e2d8482c58cfed66f1c: Bootstrap starting.
I20260925 16:33:52.251441 11142 tablet_bootstrap.cc:492] T 63d0cf6f83ea46dbb50c03192a5da0d2 P a091ffa1fa3e4a2787ceca008d229dde: Bootstrap starting.
I20260925 16:33:52.252383 11141 tablet_bootstrap.cc:654] T 63d0cf6f83ea46dbb50c03192a5da0d2 P 0069902798724e2d8482c58cfed66f1c: Neither blocks nor log segments found. Creating new log.
I20260925 16:33:52.253479 11141 log.cc:824] T 63d0cf6f83ea46dbb50c03192a5da0d2 P 0069902798724e2d8482c58cfed66f1c: Log is configured to *not* fsync() on all Append() calls
I20260925 16:33:52.253515 11142 tablet_bootstrap.cc:654] T 63d0cf6f83ea46dbb50c03192a5da0d2 P a091ffa1fa3e4a2787ceca008d229dde: Neither blocks nor log segments found. Creating new log.
I20260925 16:33:52.254153 11142 log.cc:824] T 63d0cf6f83ea46dbb50c03192a5da0d2 P a091ffa1fa3e4a2787ceca008d229dde: Log is configured to *not* fsync() on all Append() calls
I20260925 16:33:52.254908 11143 tablet_bootstrap.cc:492] T 63d0cf6f83ea46dbb50c03192a5da0d2 P 8ac8324364404f14a4f3114b0cd21ebd: Bootstrap starting.
I20260925 16:33:52.255805 11141 tablet_bootstrap.cc:492] T 63d0cf6f83ea46dbb50c03192a5da0d2 P 0069902798724e2d8482c58cfed66f1c: No bootstrap required, opened a new log
I20260925 16:33:52.256013 11141 ts_tablet_manager.cc:1397] T 63d0cf6f83ea46dbb50c03192a5da0d2 P 0069902798724e2d8482c58cfed66f1c: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 16:33:52.256093 11142 tablet_bootstrap.cc:492] T 63d0cf6f83ea46dbb50c03192a5da0d2 P a091ffa1fa3e4a2787ceca008d229dde: No bootstrap required, opened a new log
I20260925 16:33:52.256263 11142 ts_tablet_manager.cc:1397] T 63d0cf6f83ea46dbb50c03192a5da0d2 P a091ffa1fa3e4a2787ceca008d229dde: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 16:33:52.256733 11143 tablet_bootstrap.cc:654] T 63d0cf6f83ea46dbb50c03192a5da0d2 P 8ac8324364404f14a4f3114b0cd21ebd: Neither blocks nor log segments found. Creating new log.
I20260925 16:33:52.257719 11143 log.cc:824] T 63d0cf6f83ea46dbb50c03192a5da0d2 P 8ac8324364404f14a4f3114b0cd21ebd: Log is configured to *not* fsync() on all Append() calls
I20260925 16:33:52.259363 11143 tablet_bootstrap.cc:492] T 63d0cf6f83ea46dbb50c03192a5da0d2 P 8ac8324364404f14a4f3114b0cd21ebd: No bootstrap required, opened a new log
I20260925 16:33:52.259552 11143 ts_tablet_manager.cc:1397] T 63d0cf6f83ea46dbb50c03192a5da0d2 P 8ac8324364404f14a4f3114b0cd21ebd: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 16:33:52.260007 11141 raft_consensus.cc:348] T 63d0cf6f83ea46dbb50c03192a5da0d2 P 0069902798724e2d8482c58cfed66f1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac8324364404f14a4f3114b0cd21ebd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36693 } } peers { permanent_uuid: "0069902798724e2d8482c58cfed66f1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46429 } } peers { permanent_uuid: "a091ffa1fa3e4a2787ceca008d229dde" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43229 } }
I20260925 16:33:52.260007 11142 raft_consensus.cc:348] T 63d0cf6f83ea46dbb50c03192a5da0d2 P a091ffa1fa3e4a2787ceca008d229dde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac8324364404f14a4f3114b0cd21ebd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36693 } } peers { permanent_uuid: "0069902798724e2d8482c58cfed66f1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46429 } } peers { permanent_uuid: "a091ffa1fa3e4a2787ceca008d229dde" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43229 } }
I20260925 16:33:52.260279 11142 raft_consensus.cc:374] T 63d0cf6f83ea46dbb50c03192a5da0d2 P a091ffa1fa3e4a2787ceca008d229dde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:33:52.260279 11141 raft_consensus.cc:374] T 63d0cf6f83ea46dbb50c03192a5da0d2 P 0069902798724e2d8482c58cfed66f1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:33:52.260339 11141 raft_consensus.cc:729] T 63d0cf6f83ea46dbb50c03192a5da0d2 P 0069902798724e2d8482c58cfed66f1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0069902798724e2d8482c58cfed66f1c, State: Initialized, Role: FOLLOWER
I20260925 16:33:52.260347 11142 raft_consensus.cc:729] T 63d0cf6f83ea46dbb50c03192a5da0d2 P a091ffa1fa3e4a2787ceca008d229dde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a091ffa1fa3e4a2787ceca008d229dde, State: Initialized, Role: FOLLOWER
I20260925 16:33:52.260802 11142 consensus_queue.cc:260] T 63d0cf6f83ea46dbb50c03192a5da0d2 P a091ffa1fa3e4a2787ceca008d229dde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac8324364404f14a4f3114b0cd21ebd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36693 } } peers { permanent_uuid: "0069902798724e2d8482c58cfed66f1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46429 } } peers { permanent_uuid: "a091ffa1fa3e4a2787ceca008d229dde" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43229 } }
I20260925 16:33:52.260802 11141 consensus_queue.cc:260] T 63d0cf6f83ea46dbb50c03192a5da0d2 P 0069902798724e2d8482c58cfed66f1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac8324364404f14a4f3114b0cd21ebd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36693 } } peers { permanent_uuid: "0069902798724e2d8482c58cfed66f1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46429 } } peers { permanent_uuid: "a091ffa1fa3e4a2787ceca008d229dde" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43229 } }
I20260925 16:33:52.261770 11141 ts_tablet_manager.cc:1428] T 63d0cf6f83ea46dbb50c03192a5da0d2 P 0069902798724e2d8482c58cfed66f1c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 16:33:52.261770 11142 ts_tablet_manager.cc:1428] T 63d0cf6f83ea46dbb50c03192a5da0d2 P a091ffa1fa3e4a2787ceca008d229dde: Time spent starting tablet: real 0.005s user 0.000s sys 0.006s
I20260925 16:33:52.262086 10990 heartbeater.cc:503] Master 127.0.98.62:41653 was elected leader, sending a full tablet report...
I20260925 16:33:52.262356 10856 heartbeater.cc:503] Master 127.0.98.62:41653 was elected leader, sending a full tablet report...
I20260925 16:33:52.263059 11143 raft_consensus.cc:348] T 63d0cf6f83ea46dbb50c03192a5da0d2 P 8ac8324364404f14a4f3114b0cd21ebd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac8324364404f14a4f3114b0cd21ebd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36693 } } peers { permanent_uuid: "0069902798724e2d8482c58cfed66f1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46429 } } peers { permanent_uuid: "a091ffa1fa3e4a2787ceca008d229dde" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43229 } }
I20260925 16:33:52.263331 11143 raft_consensus.cc:374] T 63d0cf6f83ea46dbb50c03192a5da0d2 P 8ac8324364404f14a4f3114b0cd21ebd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:33:52.263396 11143 raft_consensus.cc:729] T 63d0cf6f83ea46dbb50c03192a5da0d2 P 8ac8324364404f14a4f3114b0cd21ebd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ac8324364404f14a4f3114b0cd21ebd, State: Initialized, Role: FOLLOWER
I20260925 16:33:52.264457 11143 consensus_queue.cc:260] T 63d0cf6f83ea46dbb50c03192a5da0d2 P 8ac8324364404f14a4f3114b0cd21ebd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac8324364404f14a4f3114b0cd21ebd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36693 } } peers { permanent_uuid: "0069902798724e2d8482c58cfed66f1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46429 } } peers { permanent_uuid: "a091ffa1fa3e4a2787ceca008d229dde" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43229 } }
I20260925 16:33:52.265082 11124 heartbeater.cc:503] Master 127.0.98.62:41653 was elected leader, sending a full tablet report...
I20260925 16:33:52.265429 11143 ts_tablet_manager.cc:1428] T 63d0cf6f83ea46dbb50c03192a5da0d2 P 8ac8324364404f14a4f3114b0cd21ebd: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260925 16:33:52.278856 11147 raft_consensus.cc:482] T 63d0cf6f83ea46dbb50c03192a5da0d2 P 0069902798724e2d8482c58cfed66f1c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:33:52.279007 11147 raft_consensus.cc:504] T 63d0cf6f83ea46dbb50c03192a5da0d2 P 0069902798724e2d8482c58cfed66f1c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac8324364404f14a4f3114b0cd21ebd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36693 } } peers { permanent_uuid: "0069902798724e2d8482c58cfed66f1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46429 } } peers { permanent_uuid: "a091ffa1fa3e4a2787ceca008d229dde" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43229 } }
I20260925 16:33:52.279990 11147 leader_election.cc:288] T 63d0cf6f83ea46dbb50c03192a5da0d2 P 0069902798724e2d8482c58cfed66f1c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ac8324364404f14a4f3114b0cd21ebd (127.0.98.3:36693), a091ffa1fa3e4a2787ceca008d229dde (127.0.98.2:43229)
I20260925 16:33:52.284071 11140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:52.280329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42961 (local address 127.0.98.2:43229)
0925 16:33:52.280449 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:52.280454 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:52.280812 (+ 358us) server_negotiation.cc:407] Connection header received
0925 16:33:52.280932 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:52.280936 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:52.280987 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:52.281044 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:33:52.281418 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:52.281900 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:52.283096 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:52.283517 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:52.283626 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:52.283688 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:52.283884 (+ 196us) server_negotiation.cc:299] Negotiation successful
0925 16:33:52.283954 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 16:33:52.284090 11150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:52.280204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36693 (local address 127.0.98.1:40037)
0925 16:33:52.280622 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:52.280638 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:52.280767 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:52.281111 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:52.281114 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:52.281130 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:33:52.281274 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:52.281279 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:52.282023 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:52.282028 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:52.282990 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:52.283000 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:52.283664 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:52.283668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:52.283771 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:52.283788 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 16:33:52.283892 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 16:33:52.283953 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"mutex_wait_us":64,"thread_start_us":118,"threads_started":1}
I20260925 16:33:52.284103 11139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:52.280369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40037 (local address 127.0.98.3:36693)
0925 16:33:52.280470 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:52.280477 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:52.280706 (+ 229us) server_negotiation.cc:407] Connection header received
0925 16:33:52.280936 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:52.280940 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:52.280984 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:52.281078 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:33:52.281387 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:52.281894 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:52.283138 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:52.283529 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:52.283652 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:52.283708 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:52.283931 (+ 223us) server_negotiation.cc:299] Negotiation successful
0925 16:33:52.283999 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 16:33:52.284096 11151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:52.280205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43229 (local address 127.0.98.1:42961)
0925 16:33:52.280700 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:52.280715 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:33:52.280818 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:33:52.281081 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:33:52.281084 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:33:52.281099 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:33:52.281283 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:52.281289 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:52.282021 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:52.282026 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:52.282970 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:33:52.282980 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:33:52.283641 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:33:52.283645 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:33:52.283743 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:52.283766 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 16:33:52.283836 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 16:33:52.283925 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"mutex_wait_us":294,"spinlock_wait_cycles":17024,"thread_start_us":88,"threads_started":1}
I20260925 16:33:52.284919 11078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d0cf6f83ea46dbb50c03192a5da0d2" candidate_uuid: "0069902798724e2d8482c58cfed66f1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ac8324364404f14a4f3114b0cd21ebd" is_pre_election: true
I20260925 16:33:52.284978 10944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d0cf6f83ea46dbb50c03192a5da0d2" candidate_uuid: "0069902798724e2d8482c58cfed66f1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a091ffa1fa3e4a2787ceca008d229dde" is_pre_election: true
I20260925 16:33:52.285197 11078 raft_consensus.cc:2454] T 63d0cf6f83ea46dbb50c03192a5da0d2 P 8ac8324364404f14a4f3114b0cd21ebd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0069902798724e2d8482c58cfed66f1c in term 0.
I20260925 16:33:52.285254 10944 raft_consensus.cc:2454] T 63d0cf6f83ea46dbb50c03192a5da0d2 P a091ffa1fa3e4a2787ceca008d229dde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0069902798724e2d8482c58cfed66f1c in term 0.
I20260925 16:33:52.285691 10745 leader_election.cc:302] T 63d0cf6f83ea46dbb50c03192a5da0d2 P 0069902798724e2d8482c58cfed66f1c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0069902798724e2d8482c58cfed66f1c, 8ac8324364404f14a4f3114b0cd21ebd, a091ffa1fa3e4a2787ceca008d229dde; no voters:
I20260925 16:33:52.285977 11147 raft_consensus.cc:2781] T 63d0cf6f83ea46dbb50c03192a5da0d2 P 0069902798724e2d8482c58cfed66f1c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:33:52.286048 11147 raft_consensus.cc:482] T 63d0cf6f83ea46dbb50c03192a5da0d2 P 0069902798724e2d8482c58cfed66f1c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:33:52.286080 11147 raft_consensus.cc:3037] T 63d0cf6f83ea46dbb50c03192a5da0d2 P 0069902798724e2d8482c58cfed66f1c [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:33:52.287145 11147 raft_consensus.cc:504] T 63d0cf6f83ea46dbb50c03192a5da0d2 P 0069902798724e2d8482c58cfed66f1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac8324364404f14a4f3114b0cd21ebd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36693 } } peers { permanent_uuid: "0069902798724e2d8482c58cfed66f1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46429 } } peers { permanent_uuid: "a091ffa1fa3e4a2787ceca008d229dde" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43229 } }
I20260925 16:33:52.287528 11147 leader_election.cc:288] T 63d0cf6f83ea46dbb50c03192a5da0d2 P 0069902798724e2d8482c58cfed66f1c [CANDIDATE]: Term 1 election: Requested vote from peers 8ac8324364404f14a4f3114b0cd21ebd (127.0.98.3:36693), a091ffa1fa3e4a2787ceca008d229dde (127.0.98.2:43229)
I20260925 16:33:52.287853 11078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d0cf6f83ea46dbb50c03192a5da0d2" candidate_uuid: "0069902798724e2d8482c58cfed66f1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ac8324364404f14a4f3114b0cd21ebd"
I20260925 16:33:52.287909 10944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d0cf6f83ea46dbb50c03192a5da0d2" candidate_uuid: "0069902798724e2d8482c58cfed66f1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a091ffa1fa3e4a2787ceca008d229dde"
I20260925 16:33:52.287967 11078 raft_consensus.cc:3037] T 63d0cf6f83ea46dbb50c03192a5da0d2 P 8ac8324364404f14a4f3114b0cd21ebd [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:33:52.288076 10944 raft_consensus.cc:3037] T 63d0cf6f83ea46dbb50c03192a5da0d2 P a091ffa1fa3e4a2787ceca008d229dde [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:33:52.288990 11078 raft_consensus.cc:2454] T 63d0cf6f83ea46dbb50c03192a5da0d2 P 8ac8324364404f14a4f3114b0cd21ebd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0069902798724e2d8482c58cfed66f1c in term 1.
I20260925 16:33:52.289125 10944 raft_consensus.cc:2454] T 63d0cf6f83ea46dbb50c03192a5da0d2 P a091ffa1fa3e4a2787ceca008d229dde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0069902798724e2d8482c58cfed66f1c in term 1.
I20260925 16:33:52.289268 10745 leader_election.cc:302] T 63d0cf6f83ea46dbb50c03192a5da0d2 P 0069902798724e2d8482c58cfed66f1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0069902798724e2d8482c58cfed66f1c, 8ac8324364404f14a4f3114b0cd21ebd; no voters:
I20260925 16:33:52.289453 11147 raft_consensus.cc:2781] T 63d0cf6f83ea46dbb50c03192a5da0d2 P 0069902798724e2d8482c58cfed66f1c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:33:52.289666 11147 raft_consensus.cc:686] T 63d0cf6f83ea46dbb50c03192a5da0d2 P 0069902798724e2d8482c58cfed66f1c [term 1 LEADER]: Becoming Leader. State: Replica: 0069902798724e2d8482c58cfed66f1c, State: Running, Role: LEADER
I20260925 16:33:52.290016 11147 consensus_queue.cc:237] T 63d0cf6f83ea46dbb50c03192a5da0d2 P 0069902798724e2d8482c58cfed66f1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac8324364404f14a4f3114b0cd21ebd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36693 } } peers { permanent_uuid: "0069902798724e2d8482c58cfed66f1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46429 } } peers { permanent_uuid: "a091ffa1fa3e4a2787ceca008d229dde" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43229 } }
I20260925 16:33:52.293177 10663 catalog_manager.cc:5495] T 63d0cf6f83ea46dbb50c03192a5da0d2 P 0069902798724e2d8482c58cfed66f1c reported cstate change: term changed from 0 to 1, leader changed from <none> to 0069902798724e2d8482c58cfed66f1c (127.0.98.1). New cstate: current_term: 1 leader_uuid: "0069902798724e2d8482c58cfed66f1c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac8324364404f14a4f3114b0cd21ebd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0069902798724e2d8482c58cfed66f1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a091ffa1fa3e4a2787ceca008d229dde" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43229 } health_report { overall_health: UNKNOWN } } }
I20260925 16:33:52.316749 11137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:52.313284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50162 (local address 127.0.98.1:46429)
0925 16:33:52.313412 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 16:33:52.313417 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:33:52.313431 (+ 14us) server_negotiation.cc:407] Connection header received
0925 16:33:52.313540 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:33:52.313543 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:33:52.313593 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:33:52.313702 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:33:52.313951 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:52.314452 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:52.315353 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:33:52.315565 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:33:52.315671 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:33:52.315981 (+ 310us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:33:52.316382 (+ 401us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:33:52.316514 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 16:33:52.316554 (+ 40us) server_negotiation.cc:299] Negotiation successful
0925 16:33:52.316605 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 16:33:52.322549 11078 raft_consensus.cc:1261] T 63d0cf6f83ea46dbb50c03192a5da0d2 P 8ac8324364404f14a4f3114b0cd21ebd [term 1 FOLLOWER]: Refusing update from remote peer 0069902798724e2d8482c58cfed66f1c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:33:52.322566 10944 raft_consensus.cc:1261] T 63d0cf6f83ea46dbb50c03192a5da0d2 P a091ffa1fa3e4a2787ceca008d229dde [term 1 FOLLOWER]: Refusing update from remote peer 0069902798724e2d8482c58cfed66f1c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:33:52.323336 11152 consensus_queue.cc:1035] T 63d0cf6f83ea46dbb50c03192a5da0d2 P 0069902798724e2d8482c58cfed66f1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a091ffa1fa3e4a2787ceca008d229dde" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:33:52.323544 11147 consensus_queue.cc:1035] T 63d0cf6f83ea46dbb50c03192a5da0d2 P 0069902798724e2d8482c58cfed66f1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ac8324364404f14a4f3114b0cd21ebd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:33:52.331537 11156 mvcc.cc:205] Tried to move back new op lower bound from 7333290116386177024 to 7333290116262887424. Current Snapshot: MvccSnapshot[applied={T|T < 7333290116386177024}]
I20260925 16:33:52.333885 11157 mvcc.cc:205] Tried to move back new op lower bound from 7333290116386177024 to 7333290116262887424. Current Snapshot: MvccSnapshot[applied={T|T < 7333290116386177024}]
I20260925 16:33:52.338250 10664 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39096:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:33:52.338428 10664 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39096:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:33:52.340785 10664 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 70685a793b014ad3a627bbf5e00e4ab0: Sending DeleteTablet for 3 replicas of tablet 63d0cf6f83ea46dbb50c03192a5da0d2
I20260925 16:33:52.341521 10790 tablet_service.cc:1514] Processing DeleteTablet for tablet 63d0cf6f83ea46dbb50c03192a5da0d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:33:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50158
I20260925 16:33:52.341519 11058 tablet_service.cc:1514] Processing DeleteTablet for tablet 63d0cf6f83ea46dbb50c03192a5da0d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:33:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35574
I20260925 16:33:52.341571 10924 tablet_service.cc:1514] Processing DeleteTablet for tablet 63d0cf6f83ea46dbb50c03192a5da0d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:33:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43936
I20260925 16:33:52.342008 11165 tablet_replica.cc:330] T 63d0cf6f83ea46dbb50c03192a5da0d2 P 0069902798724e2d8482c58cfed66f1c: stopping tablet replica
I20260925 16:33:52.342154 11167 tablet_replica.cc:330] T 63d0cf6f83ea46dbb50c03192a5da0d2 P a091ffa1fa3e4a2787ceca008d229dde: stopping tablet replica
I20260925 16:33:52.342481 11166 tablet_replica.cc:330] T 63d0cf6f83ea46dbb50c03192a5da0d2 P 8ac8324364404f14a4f3114b0cd21ebd: stopping tablet replica
I20260925 16:33:52.342506 11167 raft_consensus.cc:2229] T 63d0cf6f83ea46dbb50c03192a5da0d2 P a091ffa1fa3e4a2787ceca008d229dde [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:33:52.342782 11166 raft_consensus.cc:2229] T 63d0cf6f83ea46dbb50c03192a5da0d2 P 8ac8324364404f14a4f3114b0cd21ebd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:33:52.342968 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 10727
I20260925 16:33:52.344246 11166 raft_consensus.cc:2258] T 63d0cf6f83ea46dbb50c03192a5da0d2 P 8ac8324364404f14a4f3114b0cd21ebd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:33:52.345881 11166 ts_tablet_manager.cc:1905] T 63d0cf6f83ea46dbb50c03192a5da0d2 P 8ac8324364404f14a4f3114b0cd21ebd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:33:52.348173 11166 ts_tablet_manager.cc:1918] T 63d0cf6f83ea46dbb50c03192a5da0d2 P 8ac8324364404f14a4f3114b0cd21ebd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:33:52.348253 11166 log.cc:1196] T 63d0cf6f83ea46dbb50c03192a5da0d2 P 8ac8324364404f14a4f3114b0cd21ebd: Deleting WAL directory at /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353966814071-392-0/minicluster-data/ts-2/wal/wals/63d0cf6f83ea46dbb50c03192a5da0d2
I20260925 16:33:52.348330 11167 raft_consensus.cc:2258] T 63d0cf6f83ea46dbb50c03192a5da0d2 P a091ffa1fa3e4a2787ceca008d229dde [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:33:52.348639 11166 ts_tablet_manager.cc:1939] T 63d0cf6f83ea46dbb50c03192a5da0d2 P 8ac8324364404f14a4f3114b0cd21ebd: Deleting consensus metadata
W20260925 16:33:52.349326 10652 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46429 (error 108)
I20260925 16:33:52.349906 10651 catalog_manager.cc:4841] TS 8ac8324364404f14a4f3114b0cd21ebd (127.0.98.3:36693): tablet 63d0cf6f83ea46dbb50c03192a5da0d2 (table test-table [id=d0f2c02592c940498f4dd8344f04b17d]) successfully deleted
I20260925 16:33:52.350145 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 10861
I20260925 16:33:52.350131 11167 ts_tablet_manager.cc:1905] T 63d0cf6f83ea46dbb50c03192a5da0d2 P a091ffa1fa3e4a2787ceca008d229dde: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:33:52.350418 11136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:52.350046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46429 (local address 127.0.0.1:50170)
0925 16:33:52.350245 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:52.350317 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
W20260925 16:33:52.350759 10652 catalog_manager.cc:4584] TS 0069902798724e2d8482c58cfed66f1c (127.0.98.1:46429): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 63d0cf6f83ea46dbb50c03192a5da0d2: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46429: connect: Connection refused (error 111)
I20260925 16:33:52.357123 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 10995
I20260925 16:33:52.359807 11136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:33:52.357259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43229 (local address 127.0.0.1:43948)
0925 16:33:52.359680 (+ 2421us) negotiation.cc:107] Waiting for socket to connect
0925 16:33:52.359731 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:43229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2342}
I20260925 16:33:52.363771 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 10633
2026-09-25T16:33:52Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2239 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:33:52 dist-test-slave-65pf krb5kdc[11171](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:33:52 dist-test-slave-65pf krb5kdc[11171](info): set up 2 sockets
Sep 25 16:33:52 dist-test-slave-65pf krb5kdc[11171](info): commencing operation
krb5kdc: starting...
W20260925 16:33:53.399626 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/krb5kdc/test-user.keytab.
Sep 25 16:33:53 dist-test-slave-65pf krb5kdc[11171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:33:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:33:53Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260925 16:33:53.564893 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskWmh9MY/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/postgres -l logfile start
2026-09-25 16:33:56.059 UTC [11201] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 16:33:56.059 UTC [11201] LOG: listening on IPv4 address "127.0.98.20", port 49675
2026-09-25 16:33:56.063 UTC [11201] LOG: listening on Unix socket "/tmp/.s.PGSQL.49675"
2026-09-25 16:33:56.084 UTC [11204] LOG: database system was shut down at 2026-09-25 16:33:54 UTC
2026-09-25 16:33:56.088 UTC [11201] LOG: database system is ready to accept connections
I20260925 16:33:57.047493 392 mini_postgres.cc:96] Postgres bound to 49675
2026-09-25 16:33:57.053 UTC [11212] FATAL: database "slave" does not exist
127.0.98.20:49675 - accepting connections
I20260925 16:33:57.054303 392 mini_ranger.cc:161] Starting Ranger...
I20260925 16:33:57.069415 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 16:33:57.578997 392 mini_ranger.cc:87] Created ranger Postgres database
I20260925 16:33:57.579098 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/ranger-admin
2026-09-25 16:33:57,857 [I] DB FLAVOR :POSTGRES
2026-09-25 16:33:57,857 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 16:33:57,858 [I] Checking connection..
2026-09-25 16:33:57,858 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmh9MY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWmh9MY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49675/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 16:33:58,124 [I] Checking connection passed.
2026-09-25 16:33:58,124 [I] --------- Verifying version history table ---------
2026-09-25 16:33:58,124 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmh9MY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWmh9MY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49675/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 16:33:58,352 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 16:33:58,352 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmh9MY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWmh9MY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49675/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 16:33:58,618 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 16:33:58,619 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 16:33:58,619 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmh9MY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWmh9MY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49675/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWmh9MY/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 16:33:58.838 UTC [11276] WARNING: there is no transaction in progress
2026-09-25 16:33:58,844 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 16:33:58,844 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmh9MY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWmh9MY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49675/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 16:33:59,103 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 16:33:59,104 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 16:33:59,104 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmh9MY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWmh9MY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49675/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 16:33:59,346 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmh9MY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWmh9MY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49675/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 16:33:59,579 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmh9MY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWmh9MY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49675/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-65pf.c.gcp-upstream.internal','N') ;"
2026-09-25 16:33:59,804 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmh9MY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWmh9MY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49675/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 16:34:00,053 [I] Table x_portal_user does not exist in database ranger
2026-09-25 16:34:00,053 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmh9MY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWmh9MY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49675/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 16:34:00,306 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 16:34:00,307 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmh9MY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWmh9MY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49675/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 16:34:00,536 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmh9MY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWmh9MY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49675/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 16:34:00,779 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 16:34:00,779 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmh9MY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWmh9MY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49675/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWmh9MY/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 16:34:01.763 UTC [11384] WARNING: there is no transaction in progress
2026-09-25 16:34:01.778 UTC [11384] WARNING: there is no transaction in progress
2026-09-25 16:34:01.797 UTC [11384] WARNING: there is no transaction in progress
2026-09-25 16:34:01.812 UTC [11384] WARNING: there is no transaction in progress
2026-09-25 16:34:01.827 UTC [11384] WARNING: there is no transaction in progress
2026-09-25 16:34:01.846 UTC [11384] WARNING: there is no transaction in progress
2026-09-25 16:34:01.910 UTC [11384] WARNING: there is no transaction in progress
2026-09-25 16:34:01.916 UTC [11384] WARNING: there is no transaction in progress
2026-09-25 16:34:01.930 UTC [11384] WARNING: there is no transaction in progress
2026-09-25 16:34:01.940 UTC [11384] WARNING: there is no transaction in progress
2026-09-25 16:34:01.950 UTC [11384] WARNING: there is no transaction in progress
2026-09-25 16:34:01.964 UTC [11384] WARNING: there is no transaction in progress
2026-09-25 16:34:01.973 UTC [11384] WARNING: there is no transaction in progress
2026-09-25 16:34:01.979 UTC [11384] WARNING: there is no transaction in progress
2026-09-25 16:34:01.986 UTC [11384] WARNING: there is no transaction in progress
2026-09-25 16:34:01.993 UTC [11384] WARNING: there is no transaction in progress
2026-09-25 16:34:02,644 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 16:34:02,644 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmh9MY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWmh9MY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49675/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 16:34:02,863 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 16:34:02,863 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmh9MY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWmh9MY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49675/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 16:34:03,125 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 16:34:03,125 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmh9MY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWmh9MY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49675/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 16:34:03,351 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 16:34:03,351 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmh9MY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWmh9MY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49675/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 16:34:03,602 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmh9MY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWmh9MY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49675/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 16:34:03,861 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmh9MY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWmh9MY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49675/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-65pf.c.gcp-upstream.internal';"
2026-09-25 16:34:04,082 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 16:34:04,083 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmh9MY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWmh9MY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49675/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 16:34:04,321 [I] DB_PATCHES have already been applied
I20260925 16:34:04.327356 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskWmh9MY/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskWmh9MY/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskWmh9MY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWmh9MY/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 16:34:04.327420 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260925 16:34:04.331308 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:59657
Sep 25, 2026 4:34:04 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 4:34:04 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 4:34:04 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 4:34:04 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 4:34:04 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 4:34:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 4:34:05 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 16:34:05.460834 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.407s user 0.005s sys 0.005s
Sep 25, 2026 4:34:24 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 4:34:24 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 25, 2026 4:34:24 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 4:34:24 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 4:34:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 4:34:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 4:34:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 4:34:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 4:34:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 4:34:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 4:34:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 4:34:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 4:34:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 4:34:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 4:34:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 4:34:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 4:34:24 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 4:34:25 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 16:34:25.601912 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 16:34:25 dist-test-slave-65pf krb5kdc[11171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354065, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 16:34:25.703048 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 16:34:25.712904 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 16:34:26.575052 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 16:34:26.575134 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/ranger-kms
I20260925 16:34:26.575230 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:49675
2026-09-25 16:34:26,764 [I] DB FLAVOR :POSTGRES
2026-09-25 16:34:26,764 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 16:34:26,764 [I] Checking connection
2026-09-25 16:34:26,764 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmh9MY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49675/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 16:34:27,027 [I] connection success
2026-09-25 16:34:27,027 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 16:34:27,027 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 16:34:27,027 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmh9MY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49675/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 16:34:27,270 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 16:34:27,271 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 16:34:27,271 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 16:34:27,271 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmh9MY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49675/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 16:34:27,536 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 16:34:27.939035 392 mini_ranger_kms.cc:310] Created kms service
I20260925 16:34:28.109328 392 mini_ranger_kms.cc:326] Created kudu user
I20260925 16:34:28.156453 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 16:34:28 dist-test-slave-65pf krb5kdc[11171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354065, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 16:34:28.569767 392 mini_ranger_kms.cc:384] Added ranger policy
I20260925 16:34:28.569916 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskWmh9MY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWmh9MY/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskWmh9MY/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 16:34:28.569939 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260925 16:34:28.573582 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11564
I20260925 16:34:28.573729 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:43259
Sep 25, 2026 4:34:29 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 4:34:29 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 4:34:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 4:34:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 4:34:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 4:34:30 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 16:34:30.828928 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.126s user 0.005s sys 0.168s
I20260925 16:34:30.829078 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 16:34:30.829208 392 mini_ranger_kms.cc:401] 127.0.98.20:43259/kms/v1/keys
Sep 25 16:34:31 dist-test-slave-65pf krb5kdc[11171](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790354071, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 16:34:31 dist-test-slave-65pf krb5kdc[11171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354071, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 25 16:34:31 dist-test-slave-65pf krb5kdc[11171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354071, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 25 16:34:34 dist-test-slave-65pf krb5kdc[11171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354074, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 16:34:34.213771 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41977
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41977
--ranger_config_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:43259/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:34:34.340801 11615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:34:34.341075 11615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:34:34.341141 11615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:34:34.345197 11615 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:34:34.345265 11615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:34:34.345285 11615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:34:34.345324 11615 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:34:34.345360 11615 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:34:34.350983 11615 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:43259/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41977
--ranger_config_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:34:34.352087 11615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:34:34.353150 11615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 16:34:34.366604 11615 server_base.cc:1034] running on GCE node
W20260925 16:34:34.366567 11621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:34:34.366624 11623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:34:34.366962 11620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:34:34.367690 11615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:34:34.369182 11615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:34:34.370365 11615 hybrid_clock.cc:648] HybridClock initialized: now 1790354074370343 us; error 34 us; skew 500 ppm
Sep 25 16:34:34 dist-test-slave-65pf krb5kdc[11171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354074, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:34:34.373699 11615 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 16:34:34.374742 11615 webserver.cc:460] Webserver started at http://127.0.98.62:40529/ using document root <none> and password file <none>
I20260925 16:34:34.375291 11615 fs_manager.cc:359] Metadata directory not provided
I20260925 16:34:34.375346 11615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:34:34.375566 11615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 16:34:34 dist-test-slave-65pf krb5kdc[11171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354074, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 16:34:34.436192 11615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/master-0/data/instance:
uuid: "37f562ab8a76404a8570713316188ce1"
format_stamp: "Formatted at 2026-09-25 16:34:34 on dist-test-slave-65pf"
server_key: "75558814da0f60b0a5be7572930baab6"
server_key_iv: "41d1faed6cef1ee8361e642c94e7a8ad"
server_key_version: "kuduclusterkey@0"
I20260925 16:34:34.436827 11615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/master-0/wal/instance:
uuid: "37f562ab8a76404a8570713316188ce1"
format_stamp: "Formatted at 2026-09-25 16:34:34 on dist-test-slave-65pf"
server_key: "75558814da0f60b0a5be7572930baab6"
server_key_iv: "41d1faed6cef1ee8361e642c94e7a8ad"
server_key_version: "kuduclusterkey@0"
I20260925 16:34:34.440861 11615 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260925 16:34:34.458264 11630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:34:34.459779 11615 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 16:34:34.459936 11615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/master-0/wal
uuid: "37f562ab8a76404a8570713316188ce1"
format_stamp: "Formatted at 2026-09-25 16:34:34 on dist-test-slave-65pf"
server_key: "75558814da0f60b0a5be7572930baab6"
server_key_iv: "41d1faed6cef1ee8361e642c94e7a8ad"
server_key_version: "kuduclusterkey@0"
I20260925 16:34:34.460070 11615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:34:34.482291 11615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:34:34.485551 11615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:34:34.485790 11615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:34:34.494560 11615 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41977
I20260925 16:34:34.494617 11682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41977 every 8 connection(s)
I20260925 16:34:34.495623 11615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:34:34.499073 11683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:34:34.502787 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 11615
I20260925 16:34:34.502909 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/master-0/wal/instance
I20260925 16:34:34.503183 392 external_mini_cluster.cc:1428] Setting key 5f7fa23ef0254a9a8f945f58b921809c
I20260925 16:34:34.506906 11683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37f562ab8a76404a8570713316188ce1: Bootstrap starting.
I20260925 16:34:34.509546 11683 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37f562ab8a76404a8570713316188ce1: Neither blocks nor log segments found. Creating new log.
I20260925 16:34:34.510464 11683 log.cc:824] T 00000000000000000000000000000000 P 37f562ab8a76404a8570713316188ce1: Log is configured to *not* fsync() on all Append() calls
I20260925 16:34:34.512753 11683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37f562ab8a76404a8570713316188ce1: No bootstrap required, opened a new log
Sep 25 16:34:34 dist-test-slave-65pf krb5kdc[11171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354074, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:34:34.517438 11683 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37f562ab8a76404a8570713316188ce1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37f562ab8a76404a8570713316188ce1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41977 } }
I20260925 16:34:34.517750 11683 raft_consensus.cc:374] T 00000000000000000000000000000000 P 37f562ab8a76404a8570713316188ce1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:34:34.517858 11683 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37f562ab8a76404a8570713316188ce1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37f562ab8a76404a8570713316188ce1, State: Initialized, Role: FOLLOWER
I20260925 16:34:34.518409 11683 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37f562ab8a76404a8570713316188ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37f562ab8a76404a8570713316188ce1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41977 } }
I20260925 16:34:34.518601 11683 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37f562ab8a76404a8570713316188ce1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:34:34.518705 11683 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37f562ab8a76404a8570713316188ce1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:34:34.518826 11683 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37f562ab8a76404a8570713316188ce1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:34:34.520071 11683 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37f562ab8a76404a8570713316188ce1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37f562ab8a76404a8570713316188ce1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41977 } }
I20260925 16:34:34.520473 11683 leader_election.cc:302] T 00000000000000000000000000000000 P 37f562ab8a76404a8570713316188ce1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37f562ab8a76404a8570713316188ce1; no voters:
I20260925 16:34:34.520870 11683 leader_election.cc:288] T 00000000000000000000000000000000 P 37f562ab8a76404a8570713316188ce1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:34:34.521001 11688 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37f562ab8a76404a8570713316188ce1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:34:34.521263 11688 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37f562ab8a76404a8570713316188ce1 [term 1 LEADER]: Becoming Leader. State: Replica: 37f562ab8a76404a8570713316188ce1, State: Running, Role: LEADER
I20260925 16:34:34.521584 11686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:34:34.505196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43980 (local address 127.0.98.62:41977)
0925 16:34:34.505729 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0925 16:34:34.505750 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 16:34:34.505801 (+ 51us) server_negotiation.cc:407] Connection header received
0925 16:34:34.506661 (+ 860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:34:34.506687 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:34:34.507127 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:34:34.507473 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:34:34.509006 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:34:34.510631 (+ 1625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:34:34.511740 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:34:34.512473 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:34:34.512643 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:34:34.515861 (+ 3218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:34:34.515869 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:34:34.515890 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:34:34.515927 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:34:34.518036 (+ 2109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:34:34.518545 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:34:34.518551 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:34:34.518557 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:34:34.518724 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:34:34.519040 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:34:34.519046 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:34:34.519051 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:34:34.519564 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:34:34.519756 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0925 16:34:34.520079 (+ 323us) server_negotiation.cc:299] Negotiation successful
0925 16:34:34.520441 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":170,"threads_started":1}
I20260925 16:34:34.521637 11688 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37f562ab8a76404a8570713316188ce1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37f562ab8a76404a8570713316188ce1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41977 } }
I20260925 16:34:34.522212 11683 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37f562ab8a76404a8570713316188ce1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:34:34.523012 11688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37f562ab8a76404a8570713316188ce1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37f562ab8a76404a8570713316188ce1. Latest consensus state: current_term: 1 leader_uuid: "37f562ab8a76404a8570713316188ce1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37f562ab8a76404a8570713316188ce1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41977 } } }
I20260925 16:34:34.523123 11688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37f562ab8a76404a8570713316188ce1 [sys.catalog]: This master's current role is: LEADER
I20260925 16:34:34.523460 11692 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:34:34.524266 11690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37f562ab8a76404a8570713316188ce1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "37f562ab8a76404a8570713316188ce1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37f562ab8a76404a8570713316188ce1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41977 } } }
I20260925 16:34:34.524434 11690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37f562ab8a76404a8570713316188ce1 [sys.catalog]: This master's current role is: LEADER
I20260925 16:34:34.526991 11692 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:34:34.528787 11683 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 16:34:34.542543 11692 catalog_manager.cc:1347] Generated new cluster ID: 06446e64c26943f3a7483e293f18f05e
I20260925 16:34:34.542682 11692 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:34:34.562290 11692 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:34:34.564641 11692 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:34:34.574069 11692 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37f562ab8a76404a8570713316188ce1: Generated new TSK 0
I20260925 16:34:34.575223 11692 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 16:34:35.321990 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:43259/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41977
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:34:35.483433 11728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:34:35.483780 11728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:34:35.483841 11728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:34:35.489235 11728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:34:35.489377 11728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:34:35.489494 11728 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 16:34:35.497140 11728 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:43259/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:34:35.498682 11728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:34:35.500072 11728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:34:35.512432 11736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:34:35.512465 11735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:34:35.512920 11738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:34:35.513123 11728 server_base.cc:1034] running on GCE node
I20260925 16:34:35.513550 11728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:34:35.514264 11728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:34:35.521704 11728 hybrid_clock.cc:648] HybridClock initialized: now 1790354075521626 us; error 98 us; skew 500 ppm
Sep 25 16:34:35 dist-test-slave-65pf krb5kdc[11171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354075, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:34:35.526247 11728 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 16:34:35.527598 11728 webserver.cc:460] Webserver started at http://127.0.98.1:46563/ using document root <none> and password file <none>
I20260925 16:34:35.528366 11728 fs_manager.cc:359] Metadata directory not provided
I20260925 16:34:35.528488 11728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:34:35.528780 11728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 16:34:35 dist-test-slave-65pf krb5kdc[11171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354075, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 16:34:35.546275 11728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/ts-0/data/instance:
uuid: "1fae5bd9fbbd4064b20b960e9691c52e"
format_stamp: "Formatted at 2026-09-25 16:34:35 on dist-test-slave-65pf"
server_key: "7b9818d8c9675f46d18462de3f5395f1"
server_key_iv: "c57ccca475881a9bd7053d715f0b6da9"
server_key_version: "kuduclusterkey@0"
I20260925 16:34:35.547118 11728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1fae5bd9fbbd4064b20b960e9691c52e"
format_stamp: "Formatted at 2026-09-25 16:34:35 on dist-test-slave-65pf"
server_key: "7b9818d8c9675f46d18462de3f5395f1"
server_key_iv: "c57ccca475881a9bd7053d715f0b6da9"
server_key_version: "kuduclusterkey@0"
I20260925 16:34:35.552215 11728 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260925 16:34:35.575726 11745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:34:35.579753 11728 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.003s sys 0.000s
I20260925 16:34:35.579897 11728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/ts-0/wal
uuid: "1fae5bd9fbbd4064b20b960e9691c52e"
format_stamp: "Formatted at 2026-09-25 16:34:35 on dist-test-slave-65pf"
server_key: "7b9818d8c9675f46d18462de3f5395f1"
server_key_iv: "c57ccca475881a9bd7053d715f0b6da9"
server_key_version: "kuduclusterkey@0"
I20260925 16:34:35.580045 11728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:34:35.615861 11728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:34:35.619657 11728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:34:35.619923 11728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:34:35.620678 11728 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:34:35.621788 11728 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:34:35.621898 11728 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:34:35.622028 11728 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:34:35.622113 11728 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:34:35.636389 11728 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45095
I20260925 16:34:35.637591 11728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
I20260925 16:34:35.645787 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 11728
I20260925 16:34:35.645923 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/ts-0/wal/instance
I20260925 16:34:35.646242 392 external_mini_cluster.cc:1428] Setting key 51b232f2e34d756cfbae48f41579bfdb
I20260925 16:34:35.639406 11860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45095 every 8 connection(s)
Sep 25 16:34:35 dist-test-slave-65pf krb5kdc[11171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354075, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:34:35.663571 11864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:34:35.642884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34437 (local address 127.0.98.62:41977)
0925 16:34:35.643571 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0925 16:34:35.643577 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:34:35.645445 (+ 1868us) server_negotiation.cc:407] Connection header received
0925 16:34:35.645501 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:34:35.646771 (+ 1270us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:34:35.647033 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:34:35.647528 (+ 495us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:34:35.651432 (+ 3904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:34:35.652321 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:34:35.654134 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:34:35.654495 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:34:35.654577 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:34:35.659158 (+ 4581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:34:35.659162 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:34:35.659164 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:34:35.659188 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:34:35.660999 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:34:35.661601 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:34:35.661604 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:34:35.661606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:34:35.661705 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:34:35.662108 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:34:35.662112 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:34:35.662113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:34:35.662330 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:34:35.662408 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 16:34:35.663234 (+ 826us) server_negotiation.cc:299] Negotiation successful
0925 16:34:35.663371 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":87,"threads_started":1}
I20260925 16:34:35.665680 11863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:34:35.641720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41977 (local address 127.0.98.1:34437)
0925 16:34:35.643987 (+ 2267us) negotiation.cc:107] Waiting for socket to connect
0925 16:34:35.644044 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 16:34:35.645116 (+ 1072us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:34:35.650002 (+ 4886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:34:35.650012 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:34:35.650385 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:34:35.651224 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:34:35.651246 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:34:35.652502 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:34:35.652506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:34:35.653988 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:34:35.653999 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:34:35.654831 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:34:35.654835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:34:35.655044 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:34:35.657175 (+ 2131us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:34:35.657198 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:34:35.658951 (+ 1753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:34:35.661142 (+ 2191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:34:35.661148 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:34:35.661164 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:34:35.661487 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:34:35.661807 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:34:35.661810 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:34:35.661813 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:34:35.662000 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:34:35.662471 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:34:35.662478 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:34:35.662888 (+ 410us) client_negotiation.cc:769] Sending connection context
0925 16:34:35.664099 (+ 1211us) client_negotiation.cc:240] Negotiation successful
0925 16:34:35.664441 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2044,"thread_start_us":147,"threads_started":1}
I20260925 16:34:35.667412 11861 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41977
I20260925 16:34:35.667666 11861 heartbeater.cc:464] Registering TS with master...
I20260925 16:34:35.668251 11861 heartbeater.cc:511] Master 127.0.98.62:41977 requested a full tablet report, sending...
I20260925 16:34:35.670037 11647 ts_manager.cc:194] Registered new tserver with Master: 1fae5bd9fbbd4064b20b960e9691c52e (127.0.98.1:45095)
I20260925 16:34:35.671629 11647 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34437
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Sep 25 16:34:35 dist-test-slave-65pf krb5kdc[11171](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790354075, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 16:34:35.698340 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:43259/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41977
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:34:35.853605 11867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:34:35.854038 11867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:34:35.854175 11867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:34:35.859274 11867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:34:35.859360 11867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:34:35.859464 11867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 16:34:35.866909 11867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:43259/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:34:35.868034 11867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:34:35.869108 11867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:34:35.879132 11874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:34:35.879069 11876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:34:35.879272 11873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:34:35.880474 11867 server_base.cc:1034] running on GCE node
I20260925 16:34:35.880889 11867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:34:35.881562 11867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:34:35.882737 11867 hybrid_clock.cc:648] HybridClock initialized: now 1790354075882705 us; error 43 us; skew 500 ppm
Sep 25 16:34:35 dist-test-slave-65pf krb5kdc[11171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354075, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:34:35.885692 11867 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 16:34:35.886749 11867 webserver.cc:460] Webserver started at http://127.0.98.2:37381/ using document root <none> and password file <none>
I20260925 16:34:35.887398 11867 fs_manager.cc:359] Metadata directory not provided
I20260925 16:34:35.887483 11867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:34:35.887698 11867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 16:34:35 dist-test-slave-65pf krb5kdc[11171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354075, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 16:34:35.901350 11867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/ts-1/data/instance:
uuid: "1b0e74f16a8446f7bd1c79a6040df7e1"
format_stamp: "Formatted at 2026-09-25 16:34:35 on dist-test-slave-65pf"
server_key: "815d4581dd626141ba7ef6fc8dd566ec"
server_key_iv: "cc7ecf5197226cf107518d09997e17eb"
server_key_version: "kuduclusterkey@0"
I20260925 16:34:35.902192 11867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1b0e74f16a8446f7bd1c79a6040df7e1"
format_stamp: "Formatted at 2026-09-25 16:34:35 on dist-test-slave-65pf"
server_key: "815d4581dd626141ba7ef6fc8dd566ec"
server_key_iv: "cc7ecf5197226cf107518d09997e17eb"
server_key_version: "kuduclusterkey@0"
I20260925 16:34:35.908699 11867 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.000s
I20260925 16:34:35.925628 11883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:34:35.927372 11867 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260925 16:34:35.927542 11867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/ts-1/wal
uuid: "1b0e74f16a8446f7bd1c79a6040df7e1"
format_stamp: "Formatted at 2026-09-25 16:34:35 on dist-test-slave-65pf"
server_key: "815d4581dd626141ba7ef6fc8dd566ec"
server_key_iv: "cc7ecf5197226cf107518d09997e17eb"
server_key_version: "kuduclusterkey@0"
I20260925 16:34:35.927703 11867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:34:35.954844 11867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:34:35.957676 11867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:34:35.957825 11867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:34:35.958479 11867 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:34:35.959412 11867 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:34:35.959460 11867 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:34:35.959501 11867 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:34:35.959519 11867 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:34:35.971004 11867 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39413
I20260925 16:34:35.971180 11996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39413 every 8 connection(s)
I20260925 16:34:35.971983 11867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
I20260925 16:34:35.978699 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 11867
I20260925 16:34:35.978849 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/ts-1/wal/instance
I20260925 16:34:35.979127 392 external_mini_cluster.cc:1428] Setting key ab776fabf7484b6b9054dcd6a7ff4cc6
Sep 25 16:34:35 dist-test-slave-65pf krb5kdc[11171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354075, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:34:36.003574 11864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:34:35.973867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45159 (local address 127.0.98.62:41977)
0925 16:34:35.974025 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 16:34:35.974030 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:34:35.977227 (+ 3197us) server_negotiation.cc:407] Connection header received
0925 16:34:35.978217 (+ 990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:34:35.978220 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:34:35.978281 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:34:35.978384 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:34:35.982390 (+ 4006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:34:35.983324 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:34:35.984963 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:34:35.985401 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:34:35.985497 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:34:35.998855 (+ 13358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:34:35.998860 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:34:35.998863 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:34:35.998899 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:34:36.000752 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:34:36.001498 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:34:36.001503 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:34:36.001505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:34:36.001595 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:34:36.002048 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:34:36.002052 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:34:36.002054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:34:36.002315 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:34:36.002404 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 16:34:36.003232 (+ 828us) server_negotiation.cc:299] Negotiation successful
0925 16:34:36.003379 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 16:34:36.005856 11999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:34:35.974293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41977 (local address 127.0.98.2:45159)
0925 16:34:35.977045 (+ 2752us) negotiation.cc:107] Waiting for socket to connect
0925 16:34:35.977097 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 16:34:35.977996 (+ 899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:34:35.979737 (+ 1741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:34:35.979747 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:34:35.980095 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:34:35.982235 (+ 2140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:34:35.982255 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:34:35.983496 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:34:35.983500 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:34:35.984821 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:34:35.984835 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:34:35.985731 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:34:35.985738 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:34:35.986039 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:34:35.996609 (+ 10570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:34:35.996640 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:34:35.998553 (+ 1913us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:34:36.000920 (+ 2367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:34:36.000928 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:34:36.000949 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:34:36.001368 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:34:36.001711 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:34:36.001714 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:34:36.001716 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:34:36.001926 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:34:36.002476 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:34:36.002485 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:34:36.002910 (+ 425us) client_negotiation.cc:769] Sending connection context
0925 16:34:36.004321 (+ 1411us) client_negotiation.cc:240] Negotiation successful
0925 16:34:36.004701 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2548,"thread_start_us":111,"threads_started":1}
I20260925 16:34:36.007508 11997 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41977
I20260925 16:34:36.007756 11997 heartbeater.cc:464] Registering TS with master...
I20260925 16:34:36.008347 11997 heartbeater.cc:511] Master 127.0.98.62:41977 requested a full tablet report, sending...
I20260925 16:34:36.009512 11647 ts_manager.cc:194] Registered new tserver with Master: 1b0e74f16a8446f7bd1c79a6040df7e1 (127.0.98.2:39413)
I20260925 16:34:36.010269 11647 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45159
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 16:34:36.042204 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:43259/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41977
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 16:34:36 dist-test-slave-65pf krb5kdc[11171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354075, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260925 16:34:36.190647 12002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:34:36.191511 12002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:34:36.191660 12002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:34:36.198410 12002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:34:36.198613 12002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:34:36.198786 12002 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 16:34:36.207494 12002 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:43259/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:34:36.209297 12002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:34:36.210841 12002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:34:36.224757 12008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:34:36.225317 12010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:34:36.225978 12002 server_base.cc:1034] running on GCE node
W20260925 16:34:36.226306 12007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:34:36.226909 12002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:34:36.227747 12002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:34:36.229050 12002 hybrid_clock.cc:648] HybridClock initialized: now 1790354076228986 us; error 96 us; skew 500 ppm
Sep 25 16:34:36 dist-test-slave-65pf krb5kdc[11171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:34:36.238822 12002 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 16:34:36.240051 12002 webserver.cc:460] Webserver started at http://127.0.98.3:36347/ using document root <none> and password file <none>
I20260925 16:34:36.240911 12002 fs_manager.cc:359] Metadata directory not provided
I20260925 16:34:36.241066 12002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:34:36.241372 12002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 16:34:36 dist-test-slave-65pf krb5kdc[11171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 16:34:36.277351 12002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/ts-2/data/instance:
uuid: "ad17ec5e99494f4391390e7cb067556b"
format_stamp: "Formatted at 2026-09-25 16:34:36 on dist-test-slave-65pf"
server_key: "eabcfca4c6465d13f13da3b143789cde"
server_key_iv: "5c69607bef59dc3995c7a90b503ac520"
server_key_version: "kuduclusterkey@0"
I20260925 16:34:36.284085 12002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ad17ec5e99494f4391390e7cb067556b"
format_stamp: "Formatted at 2026-09-25 16:34:36 on dist-test-slave-65pf"
server_key: "eabcfca4c6465d13f13da3b143789cde"
server_key_iv: "5c69607bef59dc3995c7a90b503ac520"
server_key_version: "kuduclusterkey@0"
I20260925 16:34:36.296454 12002 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.004s sys 0.000s
I20260925 16:34:36.319796 12017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:34:36.322279 12002 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260925 16:34:36.322430 12002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/ts-2/wal
uuid: "ad17ec5e99494f4391390e7cb067556b"
format_stamp: "Formatted at 2026-09-25 16:34:36 on dist-test-slave-65pf"
server_key: "eabcfca4c6465d13f13da3b143789cde"
server_key_iv: "5c69607bef59dc3995c7a90b503ac520"
server_key_version: "kuduclusterkey@0"
I20260925 16:34:36.322589 12002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:34:36.348019 12002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:34:36.352067 12002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:34:36.352370 12002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:34:36.353144 12002 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:34:36.365408 12002 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:34:36.365559 12002 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:34:36.365679 12002 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:34:36.365752 12002 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:34:36.377467 12002 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38235
I20260925 16:34:36.379328 12002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
I20260925 16:34:36.380303 12130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38235 every 8 connection(s)
I20260925 16:34:36.381399 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 12002
I20260925 16:34:36.381531 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/ts-2/wal/instance
I20260925 16:34:36.381888 392 external_mini_cluster.cc:1428] Setting key c096d68eec6c7739db17899b6952b6f4
Sep 25 16:34:36 dist-test-slave-65pf krb5kdc[11171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:34:36.397751 11864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:34:36.382390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59063 (local address 127.0.98.62:41977)
0925 16:34:36.382537 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 16:34:36.382541 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:34:36.384211 (+ 1670us) server_negotiation.cc:407] Connection header received
0925 16:34:36.385472 (+ 1261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:34:36.385476 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:34:36.385527 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:34:36.385575 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:34:36.387241 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:34:36.388097 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:34:36.389801 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:34:36.390148 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:34:36.390224 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:34:36.393395 (+ 3171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:34:36.393399 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:34:36.393402 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:34:36.393428 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:34:36.395341 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:34:36.395989 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:34:36.395992 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:34:36.395994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:34:36.396076 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:34:36.396479 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:34:36.396482 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:34:36.396484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:34:36.396708 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:34:36.396786 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 16:34:36.397463 (+ 677us) server_negotiation.cc:299] Negotiation successful
0925 16:34:36.397586 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 16:34:36.399616 12133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:34:36.382902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41977 (local address 127.0.98.3:59063)
0925 16:34:36.384054 (+ 1152us) negotiation.cc:107] Waiting for socket to connect
0925 16:34:36.384113 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 16:34:36.385204 (+ 1091us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:34:36.386002 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:34:36.386011 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:34:36.386369 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:34:36.387065 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:34:36.387083 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:34:36.388260 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:34:36.388263 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:34:36.389663 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:34:36.389673 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:34:36.390426 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:34:36.390430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:34:36.390660 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:34:36.391426 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:34:36.391446 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:34:36.393198 (+ 1752us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:34:36.395488 (+ 2290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:34:36.395494 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:34:36.395513 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:34:36.395872 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:34:36.396181 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:34:36.396184 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:34:36.396186 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:34:36.396363 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:34:36.396849 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:34:36.396856 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:34:36.397196 (+ 340us) client_negotiation.cc:769] Sending connection context
0925 16:34:36.398195 (+ 999us) client_negotiation.cc:240] Negotiation successful
0925 16:34:36.398547 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":938,"thread_start_us":124,"threads_started":1}
I20260925 16:34:36.401089 12131 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41977
I20260925 16:34:36.401320 12131 heartbeater.cc:464] Registering TS with master...
I20260925 16:34:36.401867 12131 heartbeater.cc:511] Master 127.0.98.62:41977 requested a full tablet report, sending...
I20260925 16:34:36.403051 11647 ts_manager.cc:194] Registered new tserver with Master: ad17ec5e99494f4391390e7cb067556b (127.0.98.3:38235)
I20260925 16:34:36.403746 11647 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59063
I20260925 16:34:36.408478 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 16:34:36.420696 11864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:34:36.411090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60860 (local address 127.0.98.62:41977)
0925 16:34:36.411222 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 16:34:36.411227 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:34:36.411246 (+ 19us) server_negotiation.cc:407] Connection header received
0925 16:34:36.411397 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:34:36.411401 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:34:36.411450 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:34:36.411539 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:34:36.412347 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:34:36.413090 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:34:36.414337 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:34:36.414752 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:34:36.414874 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:34:36.416645 (+ 1771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:34:36.416650 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:34:36.416653 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:34:36.416693 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:34:36.418552 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:34:36.419088 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:34:36.419092 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:34:36.419093 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:34:36.419191 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:34:36.419495 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:34:36.419499 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:34:36.419500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:34:36.419749 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:34:36.419877 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 16:34:36.420323 (+ 446us) server_negotiation.cc:299] Negotiation successful
0925 16:34:36.420508 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 16:34:36.424546 11647 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60860:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 16:34:36.428632 11647 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:34:36.448998 12147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:34:36.439972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39413 (local address 127.0.0.1:33116)
0925 16:34:36.442179 (+ 2207us) negotiation.cc:107] Waiting for socket to connect
0925 16:34:36.442195 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:34:36.442284 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:34:36.443569 (+ 1285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:34:36.443575 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:34:36.443601 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:34:36.443756 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:34:36.443763 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:34:36.445374 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:34:36.445378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:34:36.446967 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:34:36.446978 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:34:36.448403 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:34:36.448407 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:34:36.448525 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:34:36.448566 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 16:34:36.448701 (+ 135us) client_negotiation.cc:240] Negotiation successful
0925 16:34:36.448830 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2137,"thread_start_us":105,"threads_started":1}
I20260925 16:34:36.449708 12143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:34:36.439907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45095 (local address 127.0.0.1:42772)
0925 16:34:36.441556 (+ 1649us) negotiation.cc:107] Waiting for socket to connect
0925 16:34:36.441609 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 16:34:36.441752 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:34:36.444248 (+ 2496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:34:36.444251 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:34:36.444267 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:34:36.444353 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:34:36.444359 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:34:36.446434 (+ 2075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:34:36.446438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:34:36.447591 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:34:36.447602 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:34:36.449367 (+ 1765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:34:36.449370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:34:36.449454 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:34:36.449478 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 16:34:36.449521 (+ 43us) client_negotiation.cc:240] Negotiation successful
0925 16:34:36.449582 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1459,"thread_start_us":104,"threads_started":1}
I20260925 16:34:36.450553 12146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:34:36.440772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42772 (local address 127.0.98.1:45095)
0925 16:34:36.443825 (+ 3053us) server_negotiation.cc:206] Beginning negotiation
0925 16:34:36.443832 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 16:34:36.443853 (+ 21us) server_negotiation.cc:407] Connection header received
0925 16:34:36.443919 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:34:36.443923 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:34:36.444086 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:34:36.445098 (+ 1012us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:34:36.445239 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:34:36.446310 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:34:36.447744 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:34:36.449239 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:34:36.450117 (+ 878us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:34:36.450286 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0925 16:34:36.450336 (+ 50us) server_negotiation.cc:299] Negotiation successful
0925 16:34:36.450423 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2950,"thread_start_us":126,"threads_started":1}
I20260925 16:34:36.453336 12145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:34:36.440754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33116 (local address 127.0.98.2:39413)
0925 16:34:36.443126 (+ 2372us) server_negotiation.cc:206] Beginning negotiation
0925 16:34:36.443150 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0925 16:34:36.443171 (+ 21us) server_negotiation.cc:407] Connection header received
0925 16:34:36.443241 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:34:36.443246 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:34:36.443425 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:34:36.443565 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:34:36.443909 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:34:36.445226 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:34:36.447151 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:34:36.448042 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:34:36.448162 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:34:36.448332 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 16:34:36.453076 (+ 4744us) server_negotiation.cc:299] Negotiation successful
0925 16:34:36.453194 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2277,"thread_start_us":85,"threads_started":1}
I20260925 16:34:36.454080 11794 tablet_service.cc:1467] Processing CreateTablet for tablet 911a6b942e8f484f9b890fec787c94b3 (DEFAULT_TABLE table=test-table [id=f3d4051b7b86416f8fca8db209bf3aed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:34:36.455204 12144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:34:36.441900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38235 (local address 127.0.0.1:35890)
0925 16:34:36.442818 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0925 16:34:36.442831 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:34:36.442921 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:34:36.450025 (+ 7104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:34:36.450029 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:34:36.450050 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:34:36.450149 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:34:36.450156 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:34:36.451709 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:34:36.451712 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:34:36.452901 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:34:36.452913 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:34:36.454774 (+ 1861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:34:36.454778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:34:36.454868 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:34:36.454892 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 16:34:36.454962 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 16:34:36.455049 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":866,"thread_start_us":97,"threads_started":1}
I20260925 16:34:36.455418 11794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 911a6b942e8f484f9b890fec787c94b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:34:36.456161 12148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:34:36.442315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35890 (local address 127.0.98.3:38235)
0925 16:34:36.442697 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0925 16:34:36.442710 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 16:34:36.442930 (+ 220us) server_negotiation.cc:407] Connection header received
0925 16:34:36.443019 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:34:36.443025 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:34:36.443255 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:34:36.443426 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:34:36.450303 (+ 6877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:34:36.451548 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:34:36.453622 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:34:36.454448 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:34:36.454558 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:34:36.455858 (+ 1300us) server_negotiation.cc:1035] Waiting for connection context
0925 16:34:36.455914 (+ 56us) server_negotiation.cc:299] Negotiation successful
0925 16:34:36.456008 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":97,"threads_started":1}
I20260925 16:34:36.456240 11931 tablet_service.cc:1467] Processing CreateTablet for tablet 911a6b942e8f484f9b890fec787c94b3 (DEFAULT_TABLE table=test-table [id=f3d4051b7b86416f8fca8db209bf3aed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:34:36.457324 11931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 911a6b942e8f484f9b890fec787c94b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:34:36.459591 12065 tablet_service.cc:1467] Processing CreateTablet for tablet 911a6b942e8f484f9b890fec787c94b3 (DEFAULT_TABLE table=test-table [id=f3d4051b7b86416f8fca8db209bf3aed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:34:36.460638 12065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 911a6b942e8f484f9b890fec787c94b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:34:36.467881 12149 tablet_bootstrap.cc:492] T 911a6b942e8f484f9b890fec787c94b3 P 1b0e74f16a8446f7bd1c79a6040df7e1: Bootstrap starting.
I20260925 16:34:36.469107 12150 tablet_bootstrap.cc:492] T 911a6b942e8f484f9b890fec787c94b3 P 1fae5bd9fbbd4064b20b960e9691c52e: Bootstrap starting.
I20260925 16:34:36.470904 12151 tablet_bootstrap.cc:492] T 911a6b942e8f484f9b890fec787c94b3 P ad17ec5e99494f4391390e7cb067556b: Bootstrap starting.
I20260925 16:34:36.470973 12149 tablet_bootstrap.cc:654] T 911a6b942e8f484f9b890fec787c94b3 P 1b0e74f16a8446f7bd1c79a6040df7e1: Neither blocks nor log segments found. Creating new log.
I20260925 16:34:36.472429 12149 log.cc:824] T 911a6b942e8f484f9b890fec787c94b3 P 1b0e74f16a8446f7bd1c79a6040df7e1: Log is configured to *not* fsync() on all Append() calls
I20260925 16:34:36.473762 12151 tablet_bootstrap.cc:654] T 911a6b942e8f484f9b890fec787c94b3 P ad17ec5e99494f4391390e7cb067556b: Neither blocks nor log segments found. Creating new log.
I20260925 16:34:36.474542 12149 tablet_bootstrap.cc:492] T 911a6b942e8f484f9b890fec787c94b3 P 1b0e74f16a8446f7bd1c79a6040df7e1: No bootstrap required, opened a new log
I20260925 16:34:36.474669 12151 log.cc:824] T 911a6b942e8f484f9b890fec787c94b3 P ad17ec5e99494f4391390e7cb067556b: Log is configured to *not* fsync() on all Append() calls
I20260925 16:34:36.475849 12149 ts_tablet_manager.cc:1397] T 911a6b942e8f484f9b890fec787c94b3 P 1b0e74f16a8446f7bd1c79a6040df7e1: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.001s
I20260925 16:34:36.476308 12150 tablet_bootstrap.cc:654] T 911a6b942e8f484f9b890fec787c94b3 P 1fae5bd9fbbd4064b20b960e9691c52e: Neither blocks nor log segments found. Creating new log.
I20260925 16:34:36.477197 12150 log.cc:824] T 911a6b942e8f484f9b890fec787c94b3 P 1fae5bd9fbbd4064b20b960e9691c52e: Log is configured to *not* fsync() on all Append() calls
I20260925 16:34:36.479460 12151 tablet_bootstrap.cc:492] T 911a6b942e8f484f9b890fec787c94b3 P ad17ec5e99494f4391390e7cb067556b: No bootstrap required, opened a new log
I20260925 16:34:36.479645 12151 ts_tablet_manager.cc:1397] T 911a6b942e8f484f9b890fec787c94b3 P ad17ec5e99494f4391390e7cb067556b: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.002s
I20260925 16:34:36.479592 12149 raft_consensus.cc:348] T 911a6b942e8f484f9b890fec787c94b3 P 1b0e74f16a8446f7bd1c79a6040df7e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fae5bd9fbbd4064b20b960e9691c52e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45095 } } peers { permanent_uuid: "1b0e74f16a8446f7bd1c79a6040df7e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39413 } } peers { permanent_uuid: "ad17ec5e99494f4391390e7cb067556b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38235 } }
I20260925 16:34:36.479846 12149 raft_consensus.cc:374] T 911a6b942e8f484f9b890fec787c94b3 P 1b0e74f16a8446f7bd1c79a6040df7e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:34:36.479892 12149 raft_consensus.cc:729] T 911a6b942e8f484f9b890fec787c94b3 P 1b0e74f16a8446f7bd1c79a6040df7e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b0e74f16a8446f7bd1c79a6040df7e1, State: Initialized, Role: FOLLOWER
I20260925 16:34:36.480551 12149 consensus_queue.cc:260] T 911a6b942e8f484f9b890fec787c94b3 P 1b0e74f16a8446f7bd1c79a6040df7e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fae5bd9fbbd4064b20b960e9691c52e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45095 } } peers { permanent_uuid: "1b0e74f16a8446f7bd1c79a6040df7e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39413 } } peers { permanent_uuid: "ad17ec5e99494f4391390e7cb067556b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38235 } }
I20260925 16:34:36.482028 12149 ts_tablet_manager.cc:1428] T 911a6b942e8f484f9b890fec787c94b3 P 1b0e74f16a8446f7bd1c79a6040df7e1: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260925 16:34:36.482425 11997 heartbeater.cc:503] Master 127.0.98.62:41977 was elected leader, sending a full tablet report...
I20260925 16:34:36.483950 12151 raft_consensus.cc:348] T 911a6b942e8f484f9b890fec787c94b3 P ad17ec5e99494f4391390e7cb067556b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fae5bd9fbbd4064b20b960e9691c52e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45095 } } peers { permanent_uuid: "1b0e74f16a8446f7bd1c79a6040df7e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39413 } } peers { permanent_uuid: "ad17ec5e99494f4391390e7cb067556b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38235 } }
I20260925 16:34:36.484151 12151 raft_consensus.cc:374] T 911a6b942e8f484f9b890fec787c94b3 P ad17ec5e99494f4391390e7cb067556b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:34:36.484189 12151 raft_consensus.cc:729] T 911a6b942e8f484f9b890fec787c94b3 P ad17ec5e99494f4391390e7cb067556b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad17ec5e99494f4391390e7cb067556b, State: Initialized, Role: FOLLOWER
I20260925 16:34:36.484596 12151 consensus_queue.cc:260] T 911a6b942e8f484f9b890fec787c94b3 P ad17ec5e99494f4391390e7cb067556b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fae5bd9fbbd4064b20b960e9691c52e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45095 } } peers { permanent_uuid: "1b0e74f16a8446f7bd1c79a6040df7e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39413 } } peers { permanent_uuid: "ad17ec5e99494f4391390e7cb067556b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38235 } }
I20260925 16:34:36.485280 12131 heartbeater.cc:503] Master 127.0.98.62:41977 was elected leader, sending a full tablet report...
I20260925 16:34:36.486186 12151 ts_tablet_manager.cc:1428] T 911a6b942e8f484f9b890fec787c94b3 P ad17ec5e99494f4391390e7cb067556b: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260925 16:34:36.487594 12150 tablet_bootstrap.cc:492] T 911a6b942e8f484f9b890fec787c94b3 P 1fae5bd9fbbd4064b20b960e9691c52e: No bootstrap required, opened a new log
I20260925 16:34:36.487869 12150 ts_tablet_manager.cc:1397] T 911a6b942e8f484f9b890fec787c94b3 P 1fae5bd9fbbd4064b20b960e9691c52e: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.000s
I20260925 16:34:36.493690 12150 raft_consensus.cc:348] T 911a6b942e8f484f9b890fec787c94b3 P 1fae5bd9fbbd4064b20b960e9691c52e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fae5bd9fbbd4064b20b960e9691c52e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45095 } } peers { permanent_uuid: "1b0e74f16a8446f7bd1c79a6040df7e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39413 } } peers { permanent_uuid: "ad17ec5e99494f4391390e7cb067556b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38235 } }
I20260925 16:34:36.493963 12150 raft_consensus.cc:374] T 911a6b942e8f484f9b890fec787c94b3 P 1fae5bd9fbbd4064b20b960e9691c52e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:34:36.494027 12150 raft_consensus.cc:729] T 911a6b942e8f484f9b890fec787c94b3 P 1fae5bd9fbbd4064b20b960e9691c52e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fae5bd9fbbd4064b20b960e9691c52e, State: Initialized, Role: FOLLOWER
I20260925 16:34:36.494539 12150 consensus_queue.cc:260] T 911a6b942e8f484f9b890fec787c94b3 P 1fae5bd9fbbd4064b20b960e9691c52e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fae5bd9fbbd4064b20b960e9691c52e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45095 } } peers { permanent_uuid: "1b0e74f16a8446f7bd1c79a6040df7e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39413 } } peers { permanent_uuid: "ad17ec5e99494f4391390e7cb067556b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38235 } }
I20260925 16:34:36.495623 12150 ts_tablet_manager.cc:1428] T 911a6b942e8f484f9b890fec787c94b3 P 1fae5bd9fbbd4064b20b960e9691c52e: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260925 16:34:36.495954 11861 heartbeater.cc:503] Master 127.0.98.62:41977 was elected leader, sending a full tablet report...
Sep 25 16:34:36 dist-test-slave-65pf krb5kdc[11171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354075, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 16:34:36.554885 12156 raft_consensus.cc:482] T 911a6b942e8f484f9b890fec787c94b3 P 1b0e74f16a8446f7bd1c79a6040df7e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:34:36.555142 12156 raft_consensus.cc:504] T 911a6b942e8f484f9b890fec787c94b3 P 1b0e74f16a8446f7bd1c79a6040df7e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fae5bd9fbbd4064b20b960e9691c52e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45095 } } peers { permanent_uuid: "1b0e74f16a8446f7bd1c79a6040df7e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39413 } } peers { permanent_uuid: "ad17ec5e99494f4391390e7cb067556b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38235 } }
I20260925 16:34:36.556604 12156 leader_election.cc:288] T 911a6b942e8f484f9b890fec787c94b3 P 1b0e74f16a8446f7bd1c79a6040df7e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1fae5bd9fbbd4064b20b960e9691c52e (127.0.98.1:45095), ad17ec5e99494f4391390e7cb067556b (127.0.98.3:38235)
I20260925 16:34:36.560365 12157 raft_consensus.cc:482] T 911a6b942e8f484f9b890fec787c94b3 P ad17ec5e99494f4391390e7cb067556b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:34:36.560575 12157 raft_consensus.cc:504] T 911a6b942e8f484f9b890fec787c94b3 P ad17ec5e99494f4391390e7cb067556b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fae5bd9fbbd4064b20b960e9691c52e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45095 } } peers { permanent_uuid: "1b0e74f16a8446f7bd1c79a6040df7e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39413 } } peers { permanent_uuid: "ad17ec5e99494f4391390e7cb067556b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38235 } }
I20260925 16:34:36.561888 12157 leader_election.cc:288] T 911a6b942e8f484f9b890fec787c94b3 P ad17ec5e99494f4391390e7cb067556b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1fae5bd9fbbd4064b20b960e9691c52e (127.0.98.1:45095), 1b0e74f16a8446f7bd1c79a6040df7e1 (127.0.98.2:39413)
I20260925 16:34:36.567929 12133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:34:36.562258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45095 (local address 127.0.98.3:33017)
0925 16:34:36.562519 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0925 16:34:36.562534 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:34:36.562641 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:34:36.563711 (+ 1070us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:34:36.563714 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:34:36.563732 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:34:36.563848 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:34:36.563854 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:34:36.565045 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:34:36.565049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:34:36.566255 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:34:36.566265 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:34:36.567535 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:34:36.567539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:34:36.567653 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:34:36.567681 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 16:34:36.567730 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 16:34:36.567798 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171}
I20260925 16:34:36.571965 12133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:34:36.563239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39413 (local address 127.0.98.3:38345)
0925 16:34:36.568081 (+ 4842us) negotiation.cc:107] Waiting for socket to connect
0925 16:34:36.568094 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:34:36.568164 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:34:36.568564 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:34:36.568567 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:34:36.568581 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:34:36.568656 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:34:36.568661 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:34:36.569567 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:34:36.569570 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:34:36.570607 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:34:36.570617 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:34:36.571548 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:34:36.571552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:34:36.571637 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:34:36.571672 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 16:34:36.571744 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 16:34:36.571808 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4794,"thread_start_us":93,"threads_started":1}
I20260925 16:34:36.572309 12145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:34:36.563586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38345 (local address 127.0.98.2:39413)
0925 16:34:36.563691 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0925 16:34:36.563696 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:34:36.568338 (+ 4642us) server_negotiation.cc:407] Connection header received
0925 16:34:36.568384 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:34:36.568387 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:34:36.568424 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:34:36.568482 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:34:36.568758 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:34:36.569434 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:34:36.570754 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:34:36.571332 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:34:36.571432 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:34:36.571493 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 16:34:36.572127 (+ 634us) server_negotiation.cc:299] Negotiation successful
0925 16:34:36.572199 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 16:34:36.572878 11951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "911a6b942e8f484f9b890fec787c94b3" candidate_uuid: "ad17ec5e99494f4391390e7cb067556b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b0e74f16a8446f7bd1c79a6040df7e1" is_pre_election: true
I20260925 16:34:36.573022 12162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:34:36.562802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33017 (local address 127.0.98.1:45095)
0925 16:34:36.563163 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0925 16:34:36.563419 (+ 256us) server_negotiation.cc:399] Waiting for connection header
0925 16:34:36.563437 (+ 18us) server_negotiation.cc:407] Connection header received
0925 16:34:36.563485 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:34:36.563487 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:34:36.563541 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:34:36.563943 (+ 402us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:34:36.564059 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:34:36.564926 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:34:36.566728 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:34:36.567401 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:34:36.572718 (+ 5317us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:34:36.572777 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 16:34:36.572813 (+ 36us) server_negotiation.cc:299] Negotiation successful
0925 16:34:36.572886 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":143,"threads_started":1}
I20260925 16:34:36.573150 11951 raft_consensus.cc:2454] T 911a6b942e8f484f9b890fec787c94b3 P 1b0e74f16a8446f7bd1c79a6040df7e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad17ec5e99494f4391390e7cb067556b in term 0.
I20260925 16:34:36.573537 12148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:34:36.557792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43123 (local address 127.0.98.3:38235)
0925 16:34:36.557910 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 16:34:36.557915 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:34:36.565308 (+ 7393us) server_negotiation.cc:407] Connection header received
0925 16:34:36.565543 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:34:36.565547 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:34:36.565592 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:34:36.565642 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:34:36.566882 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:34:36.567716 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:34:36.570875 (+ 3159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:34:36.571599 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:34:36.571705 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:34:36.571771 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 16:34:36.573351 (+ 1580us) server_negotiation.cc:299] Negotiation successful
0925 16:34:36.573432 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 16:34:36.573560 11799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "911a6b942e8f484f9b890fec787c94b3" candidate_uuid: "ad17ec5e99494f4391390e7cb067556b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fae5bd9fbbd4064b20b960e9691c52e" is_pre_election: true
I20260925 16:34:36.573728 12021 leader_election.cc:302] T 911a6b942e8f484f9b890fec787c94b3 P ad17ec5e99494f4391390e7cb067556b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b0e74f16a8446f7bd1c79a6040df7e1, ad17ec5e99494f4391390e7cb067556b; no voters:
I20260925 16:34:36.573911 11799 raft_consensus.cc:2454] T 911a6b942e8f484f9b890fec787c94b3 P 1fae5bd9fbbd4064b20b960e9691c52e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad17ec5e99494f4391390e7cb067556b in term 0.
I20260925 16:34:36.574091 12157 raft_consensus.cc:2781] T 911a6b942e8f484f9b890fec787c94b3 P ad17ec5e99494f4391390e7cb067556b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:34:36.574178 12157 raft_consensus.cc:482] T 911a6b942e8f484f9b890fec787c94b3 P ad17ec5e99494f4391390e7cb067556b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:34:36.574227 12157 raft_consensus.cc:3037] T 911a6b942e8f484f9b890fec787c94b3 P ad17ec5e99494f4391390e7cb067556b [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:34:36.574466 12160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:34:36.557430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38235 (local address 127.0.98.2:43123)
0925 16:34:36.565226 (+ 7796us) negotiation.cc:107] Waiting for socket to connect
0925 16:34:36.565241 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:34:36.565377 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:34:36.565800 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:34:36.565804 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:34:36.565822 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:34:36.565927 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:34:36.565932 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:34:36.567873 (+ 1941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:34:36.567878 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:34:36.569486 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:34:36.569499 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:34:36.571876 (+ 2377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:34:36.571880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:34:36.572005 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:34:36.572541 (+ 536us) client_negotiation.cc:769] Sending connection context
0925 16:34:36.573879 (+ 1338us) client_negotiation.cc:240] Negotiation successful
0925 16:34:36.573971 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7720,"thread_start_us":98,"threads_started":1}
I20260925 16:34:36.575127 12085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "911a6b942e8f484f9b890fec787c94b3" candidate_uuid: "1b0e74f16a8446f7bd1c79a6040df7e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad17ec5e99494f4391390e7cb067556b" is_pre_election: true
I20260925 16:34:36.575590 12157 raft_consensus.cc:504] T 911a6b942e8f484f9b890fec787c94b3 P ad17ec5e99494f4391390e7cb067556b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fae5bd9fbbd4064b20b960e9691c52e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45095 } } peers { permanent_uuid: "1b0e74f16a8446f7bd1c79a6040df7e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39413 } } peers { permanent_uuid: "ad17ec5e99494f4391390e7cb067556b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38235 } }
I20260925 16:34:36.575861 12085 raft_consensus.cc:2379] T 911a6b942e8f484f9b890fec787c94b3 P ad17ec5e99494f4391390e7cb067556b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b0e74f16a8446f7bd1c79a6040df7e1 in current term 1: Already voted for candidate ad17ec5e99494f4391390e7cb067556b in this term.
I20260925 16:34:36.575978 12157 leader_election.cc:288] T 911a6b942e8f484f9b890fec787c94b3 P ad17ec5e99494f4391390e7cb067556b [CANDIDATE]: Term 1 election: Requested vote from peers 1fae5bd9fbbd4064b20b960e9691c52e (127.0.98.1:45095), 1b0e74f16a8446f7bd1c79a6040df7e1 (127.0.98.2:39413)
I20260925 16:34:36.576924 11951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "911a6b942e8f484f9b890fec787c94b3" candidate_uuid: "ad17ec5e99494f4391390e7cb067556b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b0e74f16a8446f7bd1c79a6040df7e1"
I20260925 16:34:36.577047 11951 raft_consensus.cc:3037] T 911a6b942e8f484f9b890fec787c94b3 P 1b0e74f16a8446f7bd1c79a6040df7e1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:34:36.577082 11799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "911a6b942e8f484f9b890fec787c94b3" candidate_uuid: "ad17ec5e99494f4391390e7cb067556b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fae5bd9fbbd4064b20b960e9691c52e"
I20260925 16:34:36.577236 11799 raft_consensus.cc:3037] T 911a6b942e8f484f9b890fec787c94b3 P 1fae5bd9fbbd4064b20b960e9691c52e [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:34:36.578289 11951 raft_consensus.cc:2454] T 911a6b942e8f484f9b890fec787c94b3 P 1b0e74f16a8446f7bd1c79a6040df7e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad17ec5e99494f4391390e7cb067556b in term 1.
I20260925 16:34:36.578568 11799 raft_consensus.cc:2454] T 911a6b942e8f484f9b890fec787c94b3 P 1fae5bd9fbbd4064b20b960e9691c52e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad17ec5e99494f4391390e7cb067556b in term 1.
I20260925 16:34:36.578617 12021 leader_election.cc:302] T 911a6b942e8f484f9b890fec787c94b3 P ad17ec5e99494f4391390e7cb067556b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b0e74f16a8446f7bd1c79a6040df7e1, ad17ec5e99494f4391390e7cb067556b; no voters:
I20260925 16:34:36.578786 12157 raft_consensus.cc:2781] T 911a6b942e8f484f9b890fec787c94b3 P ad17ec5e99494f4391390e7cb067556b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:34:36.578971 12157 raft_consensus.cc:686] T 911a6b942e8f484f9b890fec787c94b3 P ad17ec5e99494f4391390e7cb067556b [term 1 LEADER]: Becoming Leader. State: Replica: ad17ec5e99494f4391390e7cb067556b, State: Running, Role: LEADER
I20260925 16:34:36.579305 12157 consensus_queue.cc:237] T 911a6b942e8f484f9b890fec787c94b3 P ad17ec5e99494f4391390e7cb067556b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fae5bd9fbbd4064b20b960e9691c52e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45095 } } peers { permanent_uuid: "1b0e74f16a8446f7bd1c79a6040df7e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39413 } } peers { permanent_uuid: "ad17ec5e99494f4391390e7cb067556b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38235 } }
I20260925 16:34:36.582947 11645 catalog_manager.cc:5495] T 911a6b942e8f484f9b890fec787c94b3 P ad17ec5e99494f4391390e7cb067556b reported cstate change: term changed from 0 to 1, leader changed from <none> to ad17ec5e99494f4391390e7cb067556b (127.0.98.3). New cstate: current_term: 1 leader_uuid: "ad17ec5e99494f4391390e7cb067556b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fae5bd9fbbd4064b20b960e9691c52e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b0e74f16a8446f7bd1c79a6040df7e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad17ec5e99494f4391390e7cb067556b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38235 } health_report { overall_health: HEALTHY } } }
I20260925 16:34:36.585813 12146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:34:36.557075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46645 (local address 127.0.98.1:45095)
0925 16:34:36.557220 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 16:34:36.557225 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:34:36.559204 (+ 1979us) server_negotiation.cc:407] Connection header received
0925 16:34:36.559420 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:34:36.559424 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:34:36.559477 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:34:36.559528 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:34:36.564229 (+ 4701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:34:36.565069 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:34:36.583591 (+ 18522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:34:36.584289 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:34:36.584381 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:34:36.584447 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 16:34:36.585586 (+ 1139us) server_negotiation.cc:299] Negotiation successful
0925 16:34:36.585679 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 16:34:36.587060 12161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:34:36.557014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45095 (local address 127.0.98.2:46645)
0925 16:34:36.559098 (+ 2084us) negotiation.cc:107] Waiting for socket to connect
0925 16:34:36.559116 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 16:34:36.559291 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:34:36.559690 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:34:36.559695 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:34:36.559718 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:34:36.564052 (+ 4334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:34:36.564061 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:34:36.565813 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:34:36.565817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:34:36.583416 (+ 17599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:34:36.583431 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:34:36.585218 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:34:36.585225 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:34:36.585414 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:34:36.585444 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 16:34:36.585969 (+ 525us) client_negotiation.cc:240] Negotiation successful
0925 16:34:36.586116 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1594,"thread_start_us":100,"threads_started":1}
I20260925 16:34:36.587491 11799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "911a6b942e8f484f9b890fec787c94b3" candidate_uuid: "1b0e74f16a8446f7bd1c79a6040df7e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fae5bd9fbbd4064b20b960e9691c52e" is_pre_election: true
I20260925 16:34:36.587697 11799 raft_consensus.cc:2379] T 911a6b942e8f484f9b890fec787c94b3 P 1fae5bd9fbbd4064b20b960e9691c52e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b0e74f16a8446f7bd1c79a6040df7e1 in current term 1: Already voted for candidate ad17ec5e99494f4391390e7cb067556b in this term.
I20260925 16:34:36.588264 11885 leader_election.cc:302] T 911a6b942e8f484f9b890fec787c94b3 P 1b0e74f16a8446f7bd1c79a6040df7e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b0e74f16a8446f7bd1c79a6040df7e1; no voters: 1fae5bd9fbbd4064b20b960e9691c52e, ad17ec5e99494f4391390e7cb067556b
I20260925 16:34:36.588759 12156 raft_consensus.cc:2726] T 911a6b942e8f484f9b890fec787c94b3 P 1b0e74f16a8446f7bd1c79a6040df7e1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 16:34:36.601719 12148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:34:36.596792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35902 (local address 127.0.98.3:38235)
0925 16:34:36.596910 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 16:34:36.596915 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:34:36.596967 (+ 52us) server_negotiation.cc:407] Connection header received
0925 16:34:36.597159 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:34:36.597162 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:34:36.597219 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:34:36.597339 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:34:36.597608 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:34:36.598352 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:34:36.599565 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:34:36.599878 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:34:36.599988 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:34:36.600420 (+ 432us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:34:36.601006 (+ 586us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:34:36.601142 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 16:34:36.601511 (+ 369us) server_negotiation.cc:299] Negotiation successful
0925 16:34:36.601572 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 16:34:36.607877 11951 raft_consensus.cc:1261] T 911a6b942e8f484f9b890fec787c94b3 P 1b0e74f16a8446f7bd1c79a6040df7e1 [term 1 FOLLOWER]: Refusing update from remote peer ad17ec5e99494f4391390e7cb067556b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:34:36.608058 11799 raft_consensus.cc:1261] T 911a6b942e8f484f9b890fec787c94b3 P 1fae5bd9fbbd4064b20b960e9691c52e [term 1 FOLLOWER]: Refusing update from remote peer ad17ec5e99494f4391390e7cb067556b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:34:36.608748 12157 consensus_queue.cc:1035] T 911a6b942e8f484f9b890fec787c94b3 P ad17ec5e99494f4391390e7cb067556b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b0e74f16a8446f7bd1c79a6040df7e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:34:36.609036 12164 consensus_queue.cc:1035] T 911a6b942e8f484f9b890fec787c94b3 P ad17ec5e99494f4391390e7cb067556b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fae5bd9fbbd4064b20b960e9691c52e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:34:36.620468 12167 mvcc.cc:205] Tried to move back new op lower bound from 7333290297777790976 to 7333290297672749056. Current Snapshot: MvccSnapshot[applied={T|T < 7333290297777790976}]
W20260925 16:34:36.644126 11862 tablet.cc:2369] T 911a6b942e8f484f9b890fec787c94b3 P 1fae5bd9fbbd4064b20b960e9691c52e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:34:36.656524 12170 mvcc.cc:205] Tried to move back new op lower bound from 7333290297777790976 to 7333290297672749056. Current Snapshot: MvccSnapshot[applied={T|T < 7333290297777790976}]
I20260925 16:34:36.659626 11646 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60860:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:34:36.659818 11646 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60860:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:34:36.663074 11646 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 37f562ab8a76404a8570713316188ce1: Sending DeleteTablet for 3 replicas of tablet 911a6b942e8f484f9b890fec787c94b3
I20260925 16:34:36.664180 11794 tablet_service.cc:1514] Processing DeleteTablet for tablet 911a6b942e8f484f9b890fec787c94b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:34:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42772
I20260925 16:34:36.665030 12065 tablet_service.cc:1514] Processing DeleteTablet for tablet 911a6b942e8f484f9b890fec787c94b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:34:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35890
I20260925 16:34:36.665607 11931 tablet_service.cc:1514] Processing DeleteTablet for tablet 911a6b942e8f484f9b890fec787c94b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:34:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33116
I20260925 16:34:36.666517 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 11728
I20260925 16:34:36.671507 12169 mvcc.cc:205] Tried to move back new op lower bound from 7333290297777790976 to 7333290297672749056. Current Snapshot: MvccSnapshot[applied={T|T < 7333290297777790976 or (T in {7333290297777790976})}]
I20260925 16:34:36.676311 12179 tablet_replica.cc:330] T 911a6b942e8f484f9b890fec787c94b3 P ad17ec5e99494f4391390e7cb067556b: stopping tablet replica
I20260925 16:34:36.676739 12179 raft_consensus.cc:2229] T 911a6b942e8f484f9b890fec787c94b3 P ad17ec5e99494f4391390e7cb067556b [term 1 LEADER]: Raft consensus shutting down.
I20260925 16:34:36.677230 12179 raft_consensus.cc:2258] T 911a6b942e8f484f9b890fec787c94b3 P ad17ec5e99494f4391390e7cb067556b [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 16:34:36.679996 11632 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45095 (error 108)
I20260925 16:34:36.686688 12179 ts_tablet_manager.cc:1905] T 911a6b942e8f484f9b890fec787c94b3 P ad17ec5e99494f4391390e7cb067556b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:34:36.686770 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 11867
I20260925 16:34:36.688617 12144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:34:36.688225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45095 (local address 127.0.0.1:42778)
0925 16:34:36.688434 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0925 16:34:36.688511 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
W20260925 16:34:36.688935 11632 catalog_manager.cc:4584] TS 1fae5bd9fbbd4064b20b960e9691c52e (127.0.98.1:45095): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 911a6b942e8f484f9b890fec787c94b3: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45095: connect: Connection refused (error 111)
I20260925 16:34:36.690413 12179 ts_tablet_manager.cc:1918] T 911a6b942e8f484f9b890fec787c94b3 P ad17ec5e99494f4391390e7cb067556b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:34:36.690552 12179 log.cc:1196] T 911a6b942e8f484f9b890fec787c94b3 P ad17ec5e99494f4391390e7cb067556b: Deleting WAL directory at /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353966814071-392-0/minicluster-data/ts-2/wal/wals/911a6b942e8f484f9b890fec787c94b3
I20260925 16:34:36.691229 12179 ts_tablet_manager.cc:1939] T 911a6b942e8f484f9b890fec787c94b3 P ad17ec5e99494f4391390e7cb067556b: Deleting consensus metadata
I20260925 16:34:36.692565 11631 catalog_manager.cc:4841] TS ad17ec5e99494f4391390e7cb067556b (127.0.98.3:38235): tablet 911a6b942e8f484f9b890fec787c94b3 (table test-table [id=f3d4051b7b86416f8fca8db209bf3aed]) successfully deleted
W20260925 16:34:36.695575 12144 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:34:36.695015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39413 (local address 127.0.0.1:33130)
0925 16:34:36.695182 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 16:34:36.695196 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:34:36.695281 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:34:36.695476 (+ 195us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:39413: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":85}
I20260925 16:34:36.696110 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 12002
I20260925 16:34:36.705506 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 11615
I20260925 16:34:36.712550 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 16:34:37.058243 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 16:34:37.058353 392 mini_ranger.cc:66] Stopping Ranger...
I20260925 16:34:37.404935 392 mini_ranger.cc:68] Stopped Ranger
2026-09-25 16:34:37.405 UTC [11201] LOG: received smart shutdown request
2026-09-25 16:34:37.408 UTC [11201] LOG: background worker "logical replication launcher" (PID 11210) exited with exit code 1
2026-09-25 16:34:37.408 UTC [11205] LOG: shutting down
2026-09-25 16:34:37.753 UTC [11201] LOG: database system is shut down
2026-09-25T16:34:37Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (45504 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:34:37 dist-test-slave-65pf krb5kdc[12190](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:34:37 dist-test-slave-65pf krb5kdc[12190](info): set up 2 sockets
Sep 25 16:34:37 dist-test-slave-65pf krb5kdc[12190](info): commencing operation
krb5kdc: starting...
W20260925 16:34:38.946748 392 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/krb5kdc/test-user.keytab.
Sep 25 16:34:39 dist-test-slave-65pf krb5kdc[12190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:34:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:34:39Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260925 16:34:39.112845 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskWmh9MY/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/postgres -l logfile start
2026-09-25 16:34:41.603 UTC [12220] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 16:34:41.603 UTC [12220] LOG: listening on IPv4 address "127.0.98.20", port 34133
2026-09-25 16:34:41.607 UTC [12220] LOG: listening on Unix socket "/tmp/.s.PGSQL.34133"
2026-09-25 16:34:41.641 UTC [12223] LOG: database system was shut down at 2026-09-25 16:34:40 UTC
2026-09-25 16:34:41.645 UTC [12220] LOG: database system is ready to accept connections
I20260925 16:34:42.586676 392 mini_postgres.cc:96] Postgres bound to 34133
2026-09-25 16:34:42.592 UTC [12231] FATAL: database "slave" does not exist
127.0.98.20:34133 - accepting connections
I20260925 16:34:42.593297 392 mini_ranger.cc:161] Starting Ranger...
I20260925 16:34:42.607677 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 16:34:43.088465 392 mini_ranger.cc:87] Created ranger Postgres database
I20260925 16:34:43.088548 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ranger-admin
2026-09-25 16:34:43,377 [I] DB FLAVOR :POSTGRES
2026-09-25 16:34:43,378 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 16:34:43,378 [I] Checking connection..
2026-09-25 16:34:43,378 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmh9MY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWmh9MY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34133/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 16:34:43,619 [I] Checking connection passed.
2026-09-25 16:34:43,619 [I] --------- Verifying version history table ---------
2026-09-25 16:34:43,620 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmh9MY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWmh9MY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34133/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 16:34:43,856 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 16:34:43,857 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmh9MY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWmh9MY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34133/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 16:34:44,105 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 16:34:44,105 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 16:34:44,105 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmh9MY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWmh9MY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34133/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWmh9MY/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 16:34:44.343 UTC [12295] WARNING: there is no transaction in progress
2026-09-25 16:34:44,350 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 16:34:44,350 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmh9MY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWmh9MY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34133/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 16:34:44,603 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 16:34:44,603 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 16:34:44,603 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmh9MY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWmh9MY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34133/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 16:34:44,844 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmh9MY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWmh9MY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34133/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 16:34:45,095 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmh9MY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWmh9MY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34133/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-65pf.c.gcp-upstream.internal','N') ;"
2026-09-25 16:34:45,327 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmh9MY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWmh9MY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34133/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 16:34:45,565 [I] Table x_portal_user does not exist in database ranger
2026-09-25 16:34:45,566 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmh9MY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWmh9MY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34133/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 16:34:45,801 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 16:34:45,801 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmh9MY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWmh9MY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34133/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 16:34:46,056 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmh9MY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWmh9MY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34133/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 16:34:46,301 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 16:34:46,302 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmh9MY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWmh9MY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34133/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWmh9MY/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 16:34:47.252 UTC [12403] WARNING: there is no transaction in progress
2026-09-25 16:34:47.269 UTC [12403] WARNING: there is no transaction in progress
2026-09-25 16:34:47.287 UTC [12403] WARNING: there is no transaction in progress
2026-09-25 16:34:47.302 UTC [12403] WARNING: there is no transaction in progress
2026-09-25 16:34:47.317 UTC [12403] WARNING: there is no transaction in progress
2026-09-25 16:34:47.334 UTC [12403] WARNING: there is no transaction in progress
2026-09-25 16:34:47.388 UTC [12403] WARNING: there is no transaction in progress
2026-09-25 16:34:47.395 UTC [12403] WARNING: there is no transaction in progress
2026-09-25 16:34:47.409 UTC [12403] WARNING: there is no transaction in progress
2026-09-25 16:34:47.419 UTC [12403] WARNING: there is no transaction in progress
2026-09-25 16:34:47.429 UTC [12403] WARNING: there is no transaction in progress
2026-09-25 16:34:47.443 UTC [12403] WARNING: there is no transaction in progress
2026-09-25 16:34:47.452 UTC [12403] WARNING: there is no transaction in progress
2026-09-25 16:34:47.458 UTC [12403] WARNING: there is no transaction in progress
2026-09-25 16:34:47.465 UTC [12403] WARNING: there is no transaction in progress
2026-09-25 16:34:47.471 UTC [12403] WARNING: there is no transaction in progress
2026-09-25 16:34:48,101 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 16:34:48,102 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmh9MY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWmh9MY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34133/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 16:34:48,327 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 16:34:48,327 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmh9MY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWmh9MY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34133/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 16:34:48,592 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 16:34:48,593 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmh9MY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWmh9MY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34133/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 16:34:48,835 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 16:34:48,836 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmh9MY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWmh9MY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34133/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 16:34:49,062 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmh9MY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWmh9MY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34133/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 16:34:49,312 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmh9MY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWmh9MY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34133/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-65pf.c.gcp-upstream.internal';"
2026-09-25 16:34:49,553 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 16:34:49,553 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmh9MY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWmh9MY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34133/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 16:34:49,780 [I] DB_PATCHES have already been applied
I20260925 16:34:49.785706 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskWmh9MY/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskWmh9MY/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskWmh9MY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWmh9MY/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 16:34:49.785772 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260925 16:34:49.789304 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:36825
Sep 25, 2026 4:34:50 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 4:34:50 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 4:34:50 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 4:34:50 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 4:34:50 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 4:34:50 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 4:34:50 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 16:34:50.889820 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.297s user 0.000s sys 0.009s
Sep 25, 2026 4:35:10 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 4:35:10 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 25, 2026 4:35:10 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 4:35:11 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 4:35:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 4:35:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 4:35:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 4:35:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 4:35:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 4:35:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 4:35:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 4:35:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 4:35:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 4:35:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 4:35:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 4:35:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 4:35:11 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 4:35:11 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 16:35:12.448513 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 16:35:12 dist-test-slave-65pf krb5kdc[12190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354112, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 16:35:12.527216 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 16:35:12.535982 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 16:35:13.372756 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 16:35:13.372845 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ranger-kms
I20260925 16:35:13.372865 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:34133
2026-09-25 16:35:13,542 [I] DB FLAVOR :POSTGRES
2026-09-25 16:35:13,543 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 16:35:13,543 [I] Checking connection
2026-09-25 16:35:13,543 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmh9MY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34133/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 16:35:13,782 [I] connection success
2026-09-25 16:35:13,782 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 16:35:13,782 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 16:35:13,782 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmh9MY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34133/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 16:35:14,042 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 16:35:14,042 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 16:35:14,042 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 16:35:14,042 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmh9MY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34133/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 16:35:14,320 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 16:35:14.794369 392 mini_ranger_kms.cc:310] Created kms service
I20260925 16:35:14.992277 392 mini_ranger_kms.cc:326] Created kudu user
I20260925 16:35:15.037891 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 16:35:15 dist-test-slave-65pf krb5kdc[12190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354112, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 16:35:15.415850 392 mini_ranger_kms.cc:384] Added ranger policy
I20260925 16:35:15.416095 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskWmh9MY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWmh9MY/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskWmh9MY/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 16:35:15.416122 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260925 16:35:15.420120 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12583
I20260925 16:35:15.420241 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:36567
Sep 25, 2026 4:35:16 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 4:35:16 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 4:35:16 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 4:35:16 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 4:35:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 4:35:17 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 16:35:17.657131 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.130s user 0.006s sys 0.147s
I20260925 16:35:17.657279 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 16:35:17.657383 392 mini_ranger_kms.cc:401] 127.0.98.20:36567/kms/v1/keys
Sep 25 16:35:17 dist-test-slave-65pf krb5kdc[12190](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790354117, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 16:35:18 dist-test-slave-65pf krb5kdc[12190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354117, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 25 16:35:18 dist-test-slave-65pf krb5kdc[12190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354117, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 25 16:35:20 dist-test-slave-65pf krb5kdc[12190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 16:35:20.997668 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41289
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41289
--ranger_config_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36567/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:35:21.108141 12634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:35:21.108500 12634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:35:21.108582 12634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:35:21.112743 12634 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:35:21.112809 12634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:35:21.112828 12634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:35:21.112845 12634 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:35:21.112860 12634 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:35:21.118506 12634 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:36567/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41289
--ranger_config_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:35:21.119658 12634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:35:21.120702 12634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:35:21.129467 12642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:35:21.129477 12639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:35:21.129679 12634 server_base.cc:1034] running on GCE node
W20260925 16:35:21.129738 12640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:35:21.130375 12634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:35:21.131637 12634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:35:21.132828 12634 hybrid_clock.cc:648] HybridClock initialized: now 1790354121132811 us; error 36 us; skew 500 ppm
Sep 25 16:35:21 dist-test-slave-65pf krb5kdc[12190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354121, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:35:21.144529 12634 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 16:35:21.145718 12634 webserver.cc:460] Webserver started at http://127.0.98.62:40229/ using document root <none> and password file <none>
I20260925 16:35:21.146335 12634 fs_manager.cc:359] Metadata directory not provided
I20260925 16:35:21.146420 12634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:35:21.146720 12634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 16:35:21 dist-test-slave-65pf krb5kdc[12190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354121, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 16:35:21.202535 12634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/master-0/data/instance:
uuid: "b459797117b049e3b92eb580851e8621"
format_stamp: "Formatted at 2026-09-25 16:35:21 on dist-test-slave-65pf"
server_key: "9248b95596c1ca7794b1f60420582b71"
server_key_iv: "ed84f68b794b0ca361149203682775d1"
server_key_version: "kuduclusterkey@0"
I20260925 16:35:21.203306 12634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/master-0/wal/instance:
uuid: "b459797117b049e3b92eb580851e8621"
format_stamp: "Formatted at 2026-09-25 16:35:21 on dist-test-slave-65pf"
server_key: "9248b95596c1ca7794b1f60420582b71"
server_key_iv: "ed84f68b794b0ca361149203682775d1"
server_key_version: "kuduclusterkey@0"
I20260925 16:35:21.207461 12634 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 16:35:21.223047 12649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:35:21.224526 12634 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 16:35:21.224648 12634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/master-0/wal
uuid: "b459797117b049e3b92eb580851e8621"
format_stamp: "Formatted at 2026-09-25 16:35:21 on dist-test-slave-65pf"
server_key: "9248b95596c1ca7794b1f60420582b71"
server_key_iv: "ed84f68b794b0ca361149203682775d1"
server_key_version: "kuduclusterkey@0"
I20260925 16:35:21.224843 12634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:35:21.252576 12634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:35:21.255651 12634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:35:21.255854 12634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:35:21.264166 12634 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41289
I20260925 16:35:21.264170 12701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41289 every 8 connection(s)
I20260925 16:35:21.265187 12634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:35:21.265707 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 12634
I20260925 16:35:21.265838 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/master-0/wal/instance
I20260925 16:35:21.266153 392 external_mini_cluster.cc:1428] Setting key b862937fbcebe05dbe9bdc2e0a72015b
I20260925 16:35:21.268599 12702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:35:21.274958 12702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b459797117b049e3b92eb580851e8621: Bootstrap starting.
Sep 25 16:35:21 dist-test-slave-65pf krb5kdc[12190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:35:21.277894 12702 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b459797117b049e3b92eb580851e8621: Neither blocks nor log segments found. Creating new log.
I20260925 16:35:21.278906 12702 log.cc:824] T 00000000000000000000000000000000 P b459797117b049e3b92eb580851e8621: Log is configured to *not* fsync() on all Append() calls
I20260925 16:35:21.281458 12702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b459797117b049e3b92eb580851e8621: No bootstrap required, opened a new log
I20260925 16:35:21.284266 12705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:21.268047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43462 (local address 127.0.98.62:41289)
0925 16:35:21.268520 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0925 16:35:21.268539 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 16:35:21.268587 (+ 48us) server_negotiation.cc:407] Connection header received
0925 16:35:21.269499 (+ 912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:35:21.269526 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:35:21.269982 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:35:21.270389 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:35:21.271335 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:21.272895 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:21.274148 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:21.274910 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:21.275095 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:21.278248 (+ 3153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:35:21.278254 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:35:21.278272 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:35:21.278303 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:35:21.280931 (+ 2628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:21.281457 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:21.281462 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:21.281470 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:21.281598 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:21.281924 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:21.281928 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:21.281929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:21.282468 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:35:21.282701 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0925 16:35:21.282998 (+ 297us) server_negotiation.cc:299] Negotiation successful
0925 16:35:21.283286 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":127,"threads_started":1}
I20260925 16:35:21.285838 12702 raft_consensus.cc:348] T 00000000000000000000000000000000 P b459797117b049e3b92eb580851e8621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b459797117b049e3b92eb580851e8621" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41289 } }
I20260925 16:35:21.286121 12702 raft_consensus.cc:374] T 00000000000000000000000000000000 P b459797117b049e3b92eb580851e8621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:35:21.286214 12702 raft_consensus.cc:729] T 00000000000000000000000000000000 P b459797117b049e3b92eb580851e8621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b459797117b049e3b92eb580851e8621, State: Initialized, Role: FOLLOWER
I20260925 16:35:21.286755 12702 consensus_queue.cc:260] T 00000000000000000000000000000000 P b459797117b049e3b92eb580851e8621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b459797117b049e3b92eb580851e8621" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41289 } }
I20260925 16:35:21.286929 12702 raft_consensus.cc:388] T 00000000000000000000000000000000 P b459797117b049e3b92eb580851e8621 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:35:21.287015 12702 raft_consensus.cc:482] T 00000000000000000000000000000000 P b459797117b049e3b92eb580851e8621 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:35:21.287112 12702 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b459797117b049e3b92eb580851e8621 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:35:21.288132 12702 raft_consensus.cc:504] T 00000000000000000000000000000000 P b459797117b049e3b92eb580851e8621 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b459797117b049e3b92eb580851e8621" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41289 } }
I20260925 16:35:21.288470 12702 leader_election.cc:302] T 00000000000000000000000000000000 P b459797117b049e3b92eb580851e8621 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b459797117b049e3b92eb580851e8621; no voters:
I20260925 16:35:21.288828 12702 leader_election.cc:288] T 00000000000000000000000000000000 P b459797117b049e3b92eb580851e8621 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:35:21.288933 12707 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b459797117b049e3b92eb580851e8621 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:35:21.289098 12707 raft_consensus.cc:686] T 00000000000000000000000000000000 P b459797117b049e3b92eb580851e8621 [term 1 LEADER]: Becoming Leader. State: Replica: b459797117b049e3b92eb580851e8621, State: Running, Role: LEADER
I20260925 16:35:21.289448 12707 consensus_queue.cc:237] T 00000000000000000000000000000000 P b459797117b049e3b92eb580851e8621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b459797117b049e3b92eb580851e8621" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41289 } }
I20260925 16:35:21.290108 12702 sys_catalog.cc:564] T 00000000000000000000000000000000 P b459797117b049e3b92eb580851e8621 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:35:21.290649 12707 sys_catalog.cc:455] T 00000000000000000000000000000000 P b459797117b049e3b92eb580851e8621 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b459797117b049e3b92eb580851e8621. Latest consensus state: current_term: 1 leader_uuid: "b459797117b049e3b92eb580851e8621" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b459797117b049e3b92eb580851e8621" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41289 } } }
I20260925 16:35:21.290778 12707 sys_catalog.cc:458] T 00000000000000000000000000000000 P b459797117b049e3b92eb580851e8621 [sys.catalog]: This master's current role is: LEADER
I20260925 16:35:21.291174 12712 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:35:21.291426 12708 sys_catalog.cc:455] T 00000000000000000000000000000000 P b459797117b049e3b92eb580851e8621 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b459797117b049e3b92eb580851e8621" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b459797117b049e3b92eb580851e8621" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41289 } } }
I20260925 16:35:21.291816 12708 sys_catalog.cc:458] T 00000000000000000000000000000000 P b459797117b049e3b92eb580851e8621 [sys.catalog]: This master's current role is: LEADER
I20260925 16:35:21.294405 12712 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:35:21.294885 12702 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 16:35:21.306193 12712 catalog_manager.cc:1347] Generated new cluster ID: a3d3c4892b414d4f8298880f851f0362
I20260925 16:35:21.306296 12712 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:35:21.345304 12712 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:35:21.347216 12712 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:35:21.358777 12712 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b459797117b049e3b92eb580851e8621: Generated new TSK 0
I20260925 16:35:21.359805 12712 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 16:35:21.982442 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36567/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41289
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:35:22.122074 12747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:35:22.122377 12747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:35:22.122478 12747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:35:22.127329 12747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:35:22.127421 12747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:35:22.127513 12747 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 16:35:22.135224 12747 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:36567/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:35:22.136796 12747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:35:22.138124 12747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:35:22.150018 12755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:35:22.150128 12757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:35:22.150398 12754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:35:22.150468 12747 server_base.cc:1034] running on GCE node
I20260925 16:35:22.150991 12747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:35:22.151788 12747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:35:22.152997 12747 hybrid_clock.cc:648] HybridClock initialized: now 1790354122152976 us; error 34 us; skew 500 ppm
Sep 25 16:35:22 dist-test-slave-65pf krb5kdc[12190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354122, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:35:22.155846 12747 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 16:35:22.156718 12747 webserver.cc:460] Webserver started at http://127.0.98.1:38935/ using document root <none> and password file <none>
I20260925 16:35:22.157343 12747 fs_manager.cc:359] Metadata directory not provided
I20260925 16:35:22.157428 12747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:35:22.157650 12747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 16:35:22 dist-test-slave-65pf krb5kdc[12190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354122, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 16:35:22.175963 12747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-0/data/instance:
uuid: "05fc8758b3a44eb3b6f8d24fe6e7057f"
format_stamp: "Formatted at 2026-09-25 16:35:22 on dist-test-slave-65pf"
server_key: "a3f6285f2e6859cf9c5e3c22bb97af86"
server_key_iv: "02e5337a6d0f5bb8d8e4601abe7145eb"
server_key_version: "kuduclusterkey@0"
I20260925 16:35:22.176720 12747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-0/wal/instance:
uuid: "05fc8758b3a44eb3b6f8d24fe6e7057f"
format_stamp: "Formatted at 2026-09-25 16:35:22 on dist-test-slave-65pf"
server_key: "a3f6285f2e6859cf9c5e3c22bb97af86"
server_key_iv: "02e5337a6d0f5bb8d8e4601abe7145eb"
server_key_version: "kuduclusterkey@0"
I20260925 16:35:22.184578 12747 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.005s sys 0.000s
I20260925 16:35:22.219394 12764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:35:22.221172 12747 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.001s sys 0.000s
I20260925 16:35:22.221324 12747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-0/wal
uuid: "05fc8758b3a44eb3b6f8d24fe6e7057f"
format_stamp: "Formatted at 2026-09-25 16:35:22 on dist-test-slave-65pf"
server_key: "a3f6285f2e6859cf9c5e3c22bb97af86"
server_key_iv: "02e5337a6d0f5bb8d8e4601abe7145eb"
server_key_version: "kuduclusterkey@0"
I20260925 16:35:22.221472 12747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:35:22.236828 12747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:35:22.240339 12747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:35:22.240543 12747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:35:22.241230 12747 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:35:22.242271 12747 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:35:22.242334 12747 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:35:22.242394 12747 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:35:22.242422 12747 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:35:22.253067 12747 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36347
I20260925 16:35:22.254273 12747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
I20260925 16:35:22.255107 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 12747
I20260925 16:35:22.255234 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-0/wal/instance
I20260925 16:35:22.255609 392 external_mini_cluster.cc:1428] Setting key 89dc0275044273e5b674160891bd85ac
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260925 16:35:22.283231 12879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36347 every 8 connection(s)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 16:35:22.290776 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36567/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41289
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 16:35:22 dist-test-slave-65pf krb5kdc[12190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354122, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:35:22.317739 12884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:22.285765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41289 (local address 127.0.98.1:40027)
0925 16:35:22.291842 (+ 6077us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:22.291912 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0925 16:35:22.293054 (+ 1142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:35:22.295384 (+ 2330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:35:22.295396 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:35:22.295773 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:35:22.296478 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:22.296498 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:22.306721 (+ 10223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:22.306726 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:22.308323 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:22.308342 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:22.309212 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:22.309216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:22.309445 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:22.310154 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:35:22.310177 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:35:22.312016 (+ 1839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:35:22.314368 (+ 2352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:35:22.314374 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:35:22.314392 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:35:22.314801 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:35:22.315140 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:35:22.315145 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:35:22.315148 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:35:22.315345 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:35:22.315814 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:35:22.315821 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:35:22.316155 (+ 334us) client_negotiation.cc:769] Sending connection context
0925 16:35:22.316365 (+ 210us) client_negotiation.cc:240] Negotiation successful
0925 16:35:22.316713 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5789,"thread_start_us":151,"threads_started":1}
I20260925 16:35:22.318550 12885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:22.286916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40027 (local address 127.0.98.62:41289)
0925 16:35:22.294771 (+ 7855us) server_negotiation.cc:206] Beginning negotiation
0925 16:35:22.294781 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 16:35:22.294799 (+ 18us) server_negotiation.cc:407] Connection header received
0925 16:35:22.294852 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:35:22.294855 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:35:22.294911 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:35:22.295014 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:35:22.296663 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:22.297535 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:22.308525 (+ 10990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:22.308896 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:22.309007 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:22.312240 (+ 3233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:35:22.312244 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:35:22.312247 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:35:22.312276 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:35:22.314205 (+ 1929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:22.314920 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:22.314924 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:22.314928 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:22.315024 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:22.315451 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:22.315455 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:22.315457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:22.315677 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:35:22.315761 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 16:35:22.318249 (+ 2488us) server_negotiation.cc:299] Negotiation successful
0925 16:35:22.318390 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7763,"thread_start_us":109,"threads_started":1}
I20260925 16:35:22.320358 12880 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41289
I20260925 16:35:22.320609 12880 heartbeater.cc:464] Registering TS with master...
I20260925 16:35:22.321177 12880 heartbeater.cc:511] Master 127.0.98.62:41289 requested a full tablet report, sending...
I20260925 16:35:22.322887 12666 ts_manager.cc:194] Registered new tserver with Master: 05fc8758b3a44eb3b6f8d24fe6e7057f (127.0.98.1:36347)
I20260925 16:35:22.324481 12666 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40027
Sep 25 16:35:22 dist-test-slave-65pf krb5kdc[12190](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790354122, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 16:35:22.460353 12886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:35:22.460810 12886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:35:22.460892 12886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:35:22.467164 12886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:35:22.467252 12886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:35:22.467351 12886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 16:35:22.475287 12886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:36567/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:35:22.476732 12886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:35:22.478122 12886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:35:22.490470 12893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:35:22.490597 12886 server_base.cc:1034] running on GCE node
W20260925 16:35:22.490958 12892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:35:22.491137 12895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:35:22.491709 12886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:35:22.492529 12886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:35:22.493757 12886 hybrid_clock.cc:648] HybridClock initialized: now 1790354122493737 us; error 46 us; skew 500 ppm
Sep 25 16:35:22 dist-test-slave-65pf krb5kdc[12190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354122, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:35:22.497331 12886 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 16:35:22.498412 12886 webserver.cc:460] Webserver started at http://127.0.98.2:36521/ using document root <none> and password file <none>
I20260925 16:35:22.499136 12886 fs_manager.cc:359] Metadata directory not provided
I20260925 16:35:22.499221 12886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:35:22.499454 12886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 16:35:22 dist-test-slave-65pf krb5kdc[12190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354122, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 16:35:22.514413 12886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-1/data/instance:
uuid: "6771b685a2284b5b9eced07cb208d017"
format_stamp: "Formatted at 2026-09-25 16:35:22 on dist-test-slave-65pf"
server_key: "0d0c2beb73075d2abdbd0599bf8af59b"
server_key_iv: "196d453fd0dd41e8c9314f511dd4c7f1"
server_key_version: "kuduclusterkey@0"
I20260925 16:35:22.515257 12886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6771b685a2284b5b9eced07cb208d017"
format_stamp: "Formatted at 2026-09-25 16:35:22 on dist-test-slave-65pf"
server_key: "0d0c2beb73075d2abdbd0599bf8af59b"
server_key_iv: "196d453fd0dd41e8c9314f511dd4c7f1"
server_key_version: "kuduclusterkey@0"
I20260925 16:35:22.520725 12886 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 16:35:22.562922 12902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:35:22.564791 12886 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.002s sys 0.000s
I20260925 16:35:22.565012 12886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-1/wal
uuid: "6771b685a2284b5b9eced07cb208d017"
format_stamp: "Formatted at 2026-09-25 16:35:22 on dist-test-slave-65pf"
server_key: "0d0c2beb73075d2abdbd0599bf8af59b"
server_key_iv: "196d453fd0dd41e8c9314f511dd4c7f1"
server_key_version: "kuduclusterkey@0"
I20260925 16:35:22.565224 12886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:35:22.581569 12886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:35:22.585366 12886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:35:22.585647 12886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:35:22.586385 12886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:35:22.587554 12886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:35:22.587663 12886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:35:22.587769 12886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:35:22.587831 12886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:35:22.611534 12886 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35017
I20260925 16:35:22.611550 13015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35017 every 8 connection(s)
I20260925 16:35:22.612643 12886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
I20260925 16:35:22.622747 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 12886
I20260925 16:35:22.622874 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-1/wal/instance
I20260925 16:35:22.623214 392 external_mini_cluster.cc:1428] Setting key 272601c1592d770097972fb395a0dfb1
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Sep 25 16:35:22 dist-test-slave-65pf krb5kdc[12190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354122, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:35:22.645704 12885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:22.616117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53595 (local address 127.0.98.62:41289)
0925 16:35:22.616252 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 16:35:22.616257 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:35:22.621974 (+ 5717us) server_negotiation.cc:407] Connection header received
0925 16:35:22.628054 (+ 6080us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:35:22.628060 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:35:22.628131 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:35:22.628192 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:35:22.630257 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:22.631194 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:22.633158 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:22.633545 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:22.633631 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:22.640332 (+ 6701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:35:22.640336 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:35:22.640339 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:35:22.640370 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:35:22.642119 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:22.642806 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:22.642810 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:22.642812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:22.642892 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:22.643277 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:22.643280 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:22.643282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:22.643509 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:35:22.643592 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 16:35:22.645383 (+ 1791us) server_negotiation.cc:299] Negotiation successful
0925 16:35:22.645548 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 16:35:22.645745 13018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:22.615275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41289 (local address 127.0.98.2:53595)
0925 16:35:22.621735 (+ 6460us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:22.621818 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0925 16:35:22.627732 (+ 5914us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:35:22.628712 (+ 980us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:35:22.628727 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:35:22.629213 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:35:22.630072 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:22.630098 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:22.631377 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:22.631380 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:22.633007 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:22.633019 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:22.633852 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:22.633856 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:22.634082 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:22.634893 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:35:22.634916 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:35:22.640091 (+ 5175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:35:22.642266 (+ 2175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:35:22.642272 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:35:22.642290 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:35:22.642689 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:35:22.642988 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:35:22.642991 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:35:22.642993 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:35:22.643177 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:35:22.643643 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:35:22.643650 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:35:22.643997 (+ 347us) client_negotiation.cc:769] Sending connection context
0925 16:35:22.644193 (+ 196us) client_negotiation.cc:240] Negotiation successful
0925 16:35:22.644547 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6164,"thread_start_us":156,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260925 16:35:22.648886 13016 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41289
I20260925 16:35:22.649135 13016 heartbeater.cc:464] Registering TS with master...
I20260925 16:35:22.649713 13016 heartbeater.cc:511] Master 127.0.98.62:41289 requested a full tablet report, sending...
I20260925 16:35:22.650767 12666 ts_manager.cc:194] Registered new tserver with Master: 6771b685a2284b5b9eced07cb208d017 (127.0.98.2:35017)
I20260925 16:35:22.651475 12666 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53595
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 16:35:22.666649 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36567/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41289
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:35:22.845449 13021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:35:22.845862 13021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:35:22.845932 13021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:35:22.852106 13021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:35:22.852195 13021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:35:22.852285 13021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 16:35:22.860255 13021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:36567/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:35:22.861693 13021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:35:22.863078 13021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:35:22.875737 13029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:35:22.876271 13027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:35:22.876881 13026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:35:22.877696 13021 server_base.cc:1034] running on GCE node
I20260925 16:35:22.878232 13021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:35:22.879112 13021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:35:22.880314 13021 hybrid_clock.cc:648] HybridClock initialized: now 1790354122880283 us; error 48 us; skew 500 ppm
Sep 25 16:35:22 dist-test-slave-65pf krb5kdc[12190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354122, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:35:22.884027 13021 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 16:35:22.885293 13021 webserver.cc:460] Webserver started at http://127.0.98.3:34699/ using document root <none> and password file <none>
I20260925 16:35:22.886149 13021 fs_manager.cc:359] Metadata directory not provided
I20260925 16:35:22.886296 13021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:35:22.886751 13021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 16:35:22 dist-test-slave-65pf krb5kdc[12190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354122, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 16:35:22.904016 13021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-2/data/instance:
uuid: "7e2574024e5242cc9484c04cdec3a2e7"
format_stamp: "Formatted at 2026-09-25 16:35:22 on dist-test-slave-65pf"
server_key: "9b38472efbf895909196dfc9f786937b"
server_key_iv: "c71759a82b76f2a8b4fb700c15dc4de5"
server_key_version: "kuduclusterkey@0"
I20260925 16:35:22.904834 13021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7e2574024e5242cc9484c04cdec3a2e7"
format_stamp: "Formatted at 2026-09-25 16:35:22 on dist-test-slave-65pf"
server_key: "9b38472efbf895909196dfc9f786937b"
server_key_iv: "c71759a82b76f2a8b4fb700c15dc4de5"
server_key_version: "kuduclusterkey@0"
I20260925 16:35:22.910738 13021 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
Sep 25 16:35:22 dist-test-slave-65pf krb5kdc[12190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354122, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 16:35:22.950512 13036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:35:22.952610 13021 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.003s sys 0.000s
I20260925 16:35:22.952795 13021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-2/wal
uuid: "7e2574024e5242cc9484c04cdec3a2e7"
format_stamp: "Formatted at 2026-09-25 16:35:22 on dist-test-slave-65pf"
server_key: "9b38472efbf895909196dfc9f786937b"
server_key_iv: "c71759a82b76f2a8b4fb700c15dc4de5"
server_key_version: "kuduclusterkey@0"
I20260925 16:35:22.952980 13021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:35:22.969065 13021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:35:22.972899 13021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:35:22.973090 13021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:35:22.973748 13021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:35:22.974833 13021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:35:22.974901 13021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:35:22.974969 13021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:35:22.975003 13021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:35:22.991043 13021 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46115
I20260925 16:35:22.992007 13149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46115 every 8 connection(s)
I20260925 16:35:22.992364 13021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
I20260925 16:35:22.997385 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 13021
I20260925 16:35:22.997498 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-2/wal/instance
I20260925 16:35:22.997781 392 external_mini_cluster.cc:1428] Setting key b1126d04d1d2bfbabbbcf5e3ddacb951
Sep 25 16:35:23 dist-test-slave-65pf krb5kdc[12190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354122, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:35:23.015426 12885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:23.001707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36853 (local address 127.0.98.62:41289)
0925 16:35:23.001834 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 16:35:23.001840 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:35:23.001855 (+ 15us) server_negotiation.cc:407] Connection header received
0925 16:35:23.002504 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:35:23.002507 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:35:23.002557 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:35:23.002629 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:35:23.004229 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:23.004982 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:23.006714 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:23.007042 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:23.007121 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:23.011117 (+ 3996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:35:23.011122 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:35:23.011125 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:35:23.011154 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:35:23.012997 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:23.013632 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:23.013636 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:23.013638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:23.013727 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:23.014120 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:23.014124 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:23.014126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:23.014364 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:35:23.014475 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 16:35:23.015121 (+ 646us) server_negotiation.cc:299] Negotiation successful
0925 16:35:23.015255 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 16:35:23.016403 13152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:23.000581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41289 (local address 127.0.98.3:36853)
0925 16:35:23.001187 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:23.001249 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0925 16:35:23.002258 (+ 1009us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:35:23.003073 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:35:23.003082 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:35:23.003431 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:35:23.004076 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:23.004094 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:23.005140 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:23.005143 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:23.006557 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:23.006569 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:23.007325 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:23.007328 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:23.007551 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:23.008358 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:35:23.008396 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:35:23.010903 (+ 2507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:35:23.013129 (+ 2226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:35:23.013136 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:35:23.013154 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:35:23.013515 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:35:23.013825 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:35:23.013828 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:35:23.013830 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:35:23.014011 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:35:23.014476 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:35:23.014483 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:35:23.014838 (+ 355us) client_negotiation.cc:769] Sending connection context
0925 16:35:23.015047 (+ 209us) client_negotiation.cc:240] Negotiation successful
0925 16:35:23.015388 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":166,"threads_started":1}
I20260925 16:35:23.017628 13150 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41289
I20260925 16:35:23.017915 13150 heartbeater.cc:464] Registering TS with master...
I20260925 16:35:23.018503 13150 heartbeater.cc:511] Master 127.0.98.62:41289 requested a full tablet report, sending...
I20260925 16:35:23.019726 12666 ts_manager.cc:194] Registered new tserver with Master: 7e2574024e5242cc9484c04cdec3a2e7 (127.0.98.3:46115)
I20260925 16:35:23.020474 12666 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36853
I20260925 16:35:23.023010 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 16:35:23.023689 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 12747
I20260925 16:35:23.028515 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 12886
I20260925 16:35:23.034371 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 13021
I20260925 16:35:23.040211 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 12634
I20260925 16:35:23.048336 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41289
--webserver_interface=127.0.98.62
--webserver_port=40229
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41289
--ranger_config_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36567/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:36567/kms with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:35:23.159248 13153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:35:23.159526 13153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:35:23.159566 13153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:35:23.163661 13153 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:35:23.163723 13153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:35:23.163743 13153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:35:23.163759 13153 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:35:23.163774 13153 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:35:23.169368 13153 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:36567/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41289
--ranger_config_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=40229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:35:23.170670 13153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:35:23.171933 13153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 16:35:23.181605 13153 server_base.cc:1034] running on GCE node
W20260925 16:35:23.181563 13159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:35:23.181560 13158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:35:23.181555 13161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:35:23.182384 13153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:35:23.183491 13153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:35:23.184674 13153 hybrid_clock.cc:648] HybridClock initialized: now 1790354123184631 us; error 55 us; skew 500 ppm
Sep 25 16:35:23 dist-test-slave-65pf krb5kdc[12190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:35:23.187589 13153 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 16:35:23.188488 13153 webserver.cc:460] Webserver started at http://127.0.98.62:40229/ using document root <none> and password file <none>
I20260925 16:35:23.189044 13153 fs_manager.cc:359] Metadata directory not provided
I20260925 16:35:23.189116 13153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 16:35:23 dist-test-slave-65pf krb5kdc[12190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 16:35:23.252686 13153 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 16:35:23.255400 13169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:35:23.256963 13153 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 16:35:23.257099 13153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/master-0/wal
uuid: "b459797117b049e3b92eb580851e8621"
format_stamp: "Formatted at 2026-09-25 16:35:21 on dist-test-slave-65pf"
server_key: "9248b95596c1ca7794b1f60420582b71"
server_key_iv: "ed84f68b794b0ca361149203682775d1"
server_key_version: "kuduclusterkey@0"
I20260925 16:35:23.257836 13153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:35:23.274304 13153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:35:23.276759 13153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:35:23.276906 13153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:35:23.283769 13153 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41289
I20260925 16:35:23.283799 13221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41289 every 8 connection(s)
I20260925 16:35:23.284818 13153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:35:23.288318 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 13153
I20260925 16:35:23.288750 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36567/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:36347
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:41289
--webserver_port=38935
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:36567/kms with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 16:35:23.289063 13222 sys_catalog.cc:263] Verifying existing consensus state
I20260925 16:35:23.290735 13222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b459797117b049e3b92eb580851e8621: Bootstrap starting.
I20260925 16:35:23.306857 13222 log.cc:824] T 00000000000000000000000000000000 P b459797117b049e3b92eb580851e8621: Log is configured to *not* fsync() on all Append() calls
I20260925 16:35:23.313117 13222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b459797117b049e3b92eb580851e8621: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 16:35:23.313519 13222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b459797117b049e3b92eb580851e8621: Bootstrap complete.
I20260925 16:35:23.318953 13222 raft_consensus.cc:348] T 00000000000000000000000000000000 P b459797117b049e3b92eb580851e8621 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b459797117b049e3b92eb580851e8621" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41289 } }
I20260925 16:35:23.319356 13222 raft_consensus.cc:729] T 00000000000000000000000000000000 P b459797117b049e3b92eb580851e8621 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b459797117b049e3b92eb580851e8621, State: Initialized, Role: FOLLOWER
I20260925 16:35:23.319821 13222 consensus_queue.cc:260] T 00000000000000000000000000000000 P b459797117b049e3b92eb580851e8621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b459797117b049e3b92eb580851e8621" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41289 } }
I20260925 16:35:23.319963 13222 raft_consensus.cc:388] T 00000000000000000000000000000000 P b459797117b049e3b92eb580851e8621 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:35:23.320053 13222 raft_consensus.cc:482] T 00000000000000000000000000000000 P b459797117b049e3b92eb580851e8621 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:35:23.320150 13222 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b459797117b049e3b92eb580851e8621 [term 1 FOLLOWER]: Advancing to term 2
I20260925 16:35:23.322017 13222 raft_consensus.cc:504] T 00000000000000000000000000000000 P b459797117b049e3b92eb580851e8621 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b459797117b049e3b92eb580851e8621" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41289 } }
I20260925 16:35:23.322351 13222 leader_election.cc:302] T 00000000000000000000000000000000 P b459797117b049e3b92eb580851e8621 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b459797117b049e3b92eb580851e8621; no voters:
I20260925 16:35:23.322727 13222 leader_election.cc:288] T 00000000000000000000000000000000 P b459797117b049e3b92eb580851e8621 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 16:35:23.322825 13226 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b459797117b049e3b92eb580851e8621 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 16:35:23.323040 13226 raft_consensus.cc:686] T 00000000000000000000000000000000 P b459797117b049e3b92eb580851e8621 [term 2 LEADER]: Becoming Leader. State: Replica: b459797117b049e3b92eb580851e8621, State: Running, Role: LEADER
I20260925 16:35:23.323391 13226 consensus_queue.cc:237] T 00000000000000000000000000000000 P b459797117b049e3b92eb580851e8621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b459797117b049e3b92eb580851e8621" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41289 } }
I20260925 16:35:23.323944 13222 sys_catalog.cc:564] T 00000000000000000000000000000000 P b459797117b049e3b92eb580851e8621 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:35:23.324355 13226 sys_catalog.cc:455] T 00000000000000000000000000000000 P b459797117b049e3b92eb580851e8621 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b459797117b049e3b92eb580851e8621. Latest consensus state: current_term: 2 leader_uuid: "b459797117b049e3b92eb580851e8621" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b459797117b049e3b92eb580851e8621" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41289 } } }
I20260925 16:35:23.324461 13226 sys_catalog.cc:458] T 00000000000000000000000000000000 P b459797117b049e3b92eb580851e8621 [sys.catalog]: This master's current role is: LEADER
I20260925 16:35:23.324817 13230 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:35:23.324864 13227 sys_catalog.cc:455] T 00000000000000000000000000000000 P b459797117b049e3b92eb580851e8621 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b459797117b049e3b92eb580851e8621" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b459797117b049e3b92eb580851e8621" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41289 } } }
I20260925 16:35:23.325124 13227 sys_catalog.cc:458] T 00000000000000000000000000000000 P b459797117b049e3b92eb580851e8621 [sys.catalog]: This master's current role is: LEADER
I20260925 16:35:23.328179 13230 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:35:23.328692 13222 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 16:35:23.328816 13230 catalog_manager.cc:1259] Loaded cluster ID: a3d3c4892b414d4f8298880f851f0362
I20260925 16:35:23.328864 13230 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:35:23.331215 13230 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:35:23.331839 13230 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b459797117b049e3b92eb580851e8621: Loaded TSK: 0
I20260925 16:35:23.333004 13230 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 16:35:23.407156 13224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:35:23.407627 13224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:35:23.407755 13224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:35:23.414656 13224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:35:23.414741 13224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:35:23.414848 13224 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 16:35:23.423395 13224 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:36567/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:36347
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=38935
--tserver_master_addrs=127.0.98.62:41289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13224
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:35:23.424753 13224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:35:23.426064 13224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:35:23.438653 13259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:35:23.438716 13262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:35:23.438967 13260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:35:23.439204 13224 server_base.cc:1034] running on GCE node
I20260925 16:35:23.439601 13224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:35:23.440358 13224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:35:23.441545 13224 hybrid_clock.cc:648] HybridClock initialized: now 1790354123441513 us; error 42 us; skew 500 ppm
Sep 25 16:35:23 dist-test-slave-65pf krb5kdc[12190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:35:23.444439 13224 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 16:35:23.445267 13224 webserver.cc:460] Webserver started at http://127.0.98.1:38935/ using document root <none> and password file <none>
I20260925 16:35:23.445916 13224 fs_manager.cc:359] Metadata directory not provided
I20260925 16:35:23.446023 13224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 16:35:23 dist-test-slave-65pf krb5kdc[12190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 16:35:23.476889 13224 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 16:35:23.479950 13270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:35:23.481801 13224 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 16:35:23.481994 13224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-0/wal
uuid: "05fc8758b3a44eb3b6f8d24fe6e7057f"
format_stamp: "Formatted at 2026-09-25 16:35:22 on dist-test-slave-65pf"
server_key: "a3f6285f2e6859cf9c5e3c22bb97af86"
server_key_iv: "02e5337a6d0f5bb8d8e4601abe7145eb"
server_key_version: "kuduclusterkey@0"
I20260925 16:35:23.482820 13224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:35:23.500900 13224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:35:23.504315 13224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:35:23.504528 13224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:35:23.505170 13224 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:35:23.506345 13224 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:35:23.506418 13224 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:35:23.506482 13224 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:35:23.506508 13224 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:35:23.517591 13224 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36347
I20260925 16:35:23.517607 13383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36347 every 8 connection(s)
I20260925 16:35:23.519078 13224 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
I20260925 16:35:23.520006 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 13224
I20260925 16:35:23.520536 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36567/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:35017
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:41289
--webserver_port=36521
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:36567/kms with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 16:35:23 dist-test-slave-65pf krb5kdc[12190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:35:23.577625 13388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:23.534846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58295 (local address 127.0.98.62:41289)
0925 16:35:23.554931 (+ 20085us) server_negotiation.cc:206] Beginning negotiation
0925 16:35:23.554958 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0925 16:35:23.555023 (+ 65us) server_negotiation.cc:407] Connection header received
0925 16:35:23.555404 (+ 381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:35:23.555424 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:35:23.555925 (+ 501us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:35:23.556354 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:35:23.557920 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:23.559653 (+ 1733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:23.561429 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:23.562178 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:23.562371 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:23.571517 (+ 9146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:35:23.571524 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:35:23.571541 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:35:23.571578 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:35:23.573630 (+ 2052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:23.574299 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:23.574305 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:23.574311 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:23.574523 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:23.574969 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:23.574972 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:23.574974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:23.575417 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:35:23.575534 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 16:35:23.576251 (+ 717us) server_negotiation.cc:299] Negotiation successful
0925 16:35:23.576576 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19852,"thread_start_us":130,"threads_started":1}
I20260925 16:35:23.579311 13387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:23.533230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41289 (local address 127.0.98.1:58295)
0925 16:35:23.539296 (+ 6066us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:23.539364 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0925 16:35:23.540544 (+ 1180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:35:23.556540 (+ 15996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:35:23.556553 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:35:23.556963 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:35:23.557658 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:23.557679 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:23.559848 (+ 2169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:23.559851 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:23.561282 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:23.561293 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:23.566799 (+ 5506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:23.566805 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:23.567010 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:23.569383 (+ 2373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:35:23.569405 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:35:23.571243 (+ 1838us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:35:23.573785 (+ 2542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:35:23.573791 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:35:23.573804 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:35:23.574154 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:35:23.574695 (+ 541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:35:23.574698 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:35:23.574700 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:35:23.574861 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:35:23.575617 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:35:23.575623 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:35:23.575903 (+ 280us) client_negotiation.cc:769] Sending connection context
0925 16:35:23.578106 (+ 2203us) client_negotiation.cc:240] Negotiation successful
0925 16:35:23.578432 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5471,"thread_start_us":194,"threads_started":1}
I20260925 16:35:23.581991 13384 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41289
I20260925 16:35:23.582229 13384 heartbeater.cc:464] Registering TS with master...
I20260925 16:35:23.582818 13384 heartbeater.cc:511] Master 127.0.98.62:41289 requested a full tablet report, sending...
W20260925 16:35:23.584972 13384 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41289 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 16:35:23.585100 13384 heartbeater.cc:464] Registering TS with master...
I20260925 16:35:23.585282 13384 heartbeater.cc:511] Master 127.0.98.62:41289 requested a full tablet report, sending...
I20260925 16:35:23.586115 13384 heartbeater.cc:464] Registering TS with master...
I20260925 16:35:23.586282 13384 heartbeater.cc:511] Master 127.0.98.62:41289 requested a full tablet report, sending...
W20260925 16:35:23.587021 13384 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 16:35:23.683647 13386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:35:23.683970 13386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:35:23.684058 13386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:35:23.688939 13386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:35:23.689036 13386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:35:23.689157 13386 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 16:35:23.696740 13386 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:36567/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:35017
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36521
--tserver_master_addrs=127.0.98.62:41289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13386
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:35:23.698117 13386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:35:23.699471 13386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:35:23.711100 13393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:35:23.711102 13394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:35:23.711367 13386 server_base.cc:1034] running on GCE node
W20260925 16:35:23.711552 13396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:35:23.712031 13386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:35:23.712739 13386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:35:23.716055 13386 hybrid_clock.cc:648] HybridClock initialized: now 1790354123716029 us; error 38 us; skew 500 ppm
Sep 25 16:35:23 dist-test-slave-65pf krb5kdc[12190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:35:23.720747 13386 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 16:35:23.721694 13386 webserver.cc:460] Webserver started at http://127.0.98.2:36521/ using document root <none> and password file <none>
I20260925 16:35:23.722250 13386 fs_manager.cc:359] Metadata directory not provided
I20260925 16:35:23.722361 13386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 16:35:23 dist-test-slave-65pf krb5kdc[12190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 16:35:23.742558 13386 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 16:35:23.745385 13404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:35:23.746930 13386 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 16:35:23.747066 13386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-1/wal
uuid: "6771b685a2284b5b9eced07cb208d017"
format_stamp: "Formatted at 2026-09-25 16:35:22 on dist-test-slave-65pf"
server_key: "0d0c2beb73075d2abdbd0599bf8af59b"
server_key_iv: "196d453fd0dd41e8c9314f511dd4c7f1"
server_key_version: "kuduclusterkey@0"
I20260925 16:35:23.747794 13386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:35:23.766671 13386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:35:23.770150 13386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:35:23.770354 13386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:35:23.771059 13386 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:35:23.772089 13386 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:35:23.772163 13386 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:35:23.772229 13386 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:35:23.772259 13386 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:35:23.783829 13386 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35017
I20260925 16:35:23.783910 13517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35017 every 8 connection(s)
I20260925 16:35:23.784904 13386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
I20260925 16:35:23.789700 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 13386
I20260925 16:35:23.790210 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36567/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:46115
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:41289
--webserver_port=34699
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:36567/kms with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 16:35:23 dist-test-slave-65pf krb5kdc[12190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:35:23.804725 13388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:23.788606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59969 (local address 127.0.98.62:41289)
0925 16:35:23.788745 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 16:35:23.788750 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:35:23.788957 (+ 207us) server_negotiation.cc:407] Connection header received
0925 16:35:23.792634 (+ 3677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:35:23.792638 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:35:23.792701 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:35:23.792804 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:35:23.794354 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:23.795248 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:23.796948 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:23.797314 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:23.797387 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:23.800436 (+ 3049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:35:23.800440 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:35:23.800443 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:35:23.800474 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:35:23.802236 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:23.802917 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:23.802921 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:23.802923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:23.803019 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:23.803405 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:23.803409 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:23.803411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:23.803623 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:35:23.803704 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 16:35:23.804409 (+ 705us) server_negotiation.cc:299] Negotiation successful
0925 16:35:23.804548 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 16:35:23.806110 13520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:23.787499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41289 (local address 127.0.98.2:59969)
0925 16:35:23.788805 (+ 1306us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:23.788859 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 16:35:23.792357 (+ 3498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:35:23.793132 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:35:23.793142 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:35:23.793497 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:35:23.794168 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:23.794188 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:23.795409 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:23.795412 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:23.796810 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:23.796820 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:23.797594 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:23.797598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:23.797788 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:23.798485 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:35:23.798504 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:35:23.800249 (+ 1745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:35:23.802429 (+ 2180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:35:23.802435 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:35:23.802452 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:35:23.802800 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:35:23.803130 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:35:23.803132 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:35:23.803135 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:35:23.803302 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:35:23.803769 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:35:23.803775 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:35:23.804106 (+ 331us) client_negotiation.cc:769] Sending connection context
0925 16:35:23.804350 (+ 244us) client_negotiation.cc:240] Negotiation successful
0925 16:35:23.804668 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1059,"thread_start_us":120,"threads_started":1}
I20260925 16:35:23.807647 13518 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41289
I20260925 16:35:23.807917 13518 heartbeater.cc:464] Registering TS with master...
I20260925 16:35:23.808400 13518 heartbeater.cc:511] Master 127.0.98.62:41289 requested a full tablet report, sending...
W20260925 16:35:23.809882 13518 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41289 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 16:35:23.810019 13518 heartbeater.cc:464] Registering TS with master...
I20260925 16:35:23.810178 13518 heartbeater.cc:511] Master 127.0.98.62:41289 requested a full tablet report, sending...
I20260925 16:35:23.811044 13518 heartbeater.cc:464] Registering TS with master...
I20260925 16:35:23.811233 13518 heartbeater.cc:511] Master 127.0.98.62:41289 requested a full tablet report, sending...
W20260925 16:35:23.811949 13518 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 16:35:23.932446 13521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:35:23.932740 13521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:35:23.932847 13521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:35:23.937001 13521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:35:23.937089 13521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:35:23.937177 13521 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 16:35:23.944316 13521 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:36567/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:46115
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=34699
--tserver_master_addrs=127.0.98.62:41289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13521
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:35:23.945498 13521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:35:23.946559 13521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:35:23.956548 13528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:35:23.956636 13530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:35:23.956962 13521 server_base.cc:1034] running on GCE node
W20260925 16:35:23.957448 13527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:35:23.958002 13521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:35:23.958863 13521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:35:23.960191 13521 hybrid_clock.cc:648] HybridClock initialized: now 1790354123960163 us; error 39 us; skew 500 ppm
Sep 25 16:35:23 dist-test-slave-65pf krb5kdc[12190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:35:23.964862 13521 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 16:35:23.966004 13521 webserver.cc:460] Webserver started at http://127.0.98.3:34699/ using document root <none> and password file <none>
I20260925 16:35:23.966741 13521 fs_manager.cc:359] Metadata directory not provided
I20260925 16:35:23.966848 13521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 16:35:23 dist-test-slave-65pf krb5kdc[12190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 16:35:23.993954 13521 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 16:35:23.998636 13538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:35:24.000191 13521 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260925 16:35:24.000326 13521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-2/wal
uuid: "7e2574024e5242cc9484c04cdec3a2e7"
format_stamp: "Formatted at 2026-09-25 16:35:22 on dist-test-slave-65pf"
server_key: "9b38472efbf895909196dfc9f786937b"
server_key_iv: "c71759a82b76f2a8b4fb700c15dc4de5"
server_key_version: "kuduclusterkey@0"
I20260925 16:35:24.001051 13521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:35:24.017486 13521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:35:24.020668 13521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:35:24.020922 13521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:35:24.021559 13521 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:35:24.022473 13521 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:35:24.022522 13521 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:35:24.022588 13521 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:35:24.022655 13521 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:35:24.033457 13521 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46115
I20260925 16:35:24.034479 13521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
I20260925 16:35:24.034802 13651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46115 every 8 connection(s)
I20260925 16:35:24.042033 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 13521
Sep 25 16:35:24 dist-test-slave-65pf krb5kdc[12190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:35:24.049045 13388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:24.036590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58895 (local address 127.0.98.62:41289)
0925 16:35:24.036749 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 16:35:24.036754 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:35:24.037533 (+ 779us) server_negotiation.cc:407] Connection header received
0925 16:35:24.038518 (+ 985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:35:24.038521 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:35:24.038573 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:35:24.038680 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:35:24.039719 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:24.040271 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:24.041724 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:24.042016 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:24.042076 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:24.045002 (+ 2926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:35:24.045006 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:35:24.045009 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:35:24.045034 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:35:24.046860 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:24.047364 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:24.047369 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:24.047371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:24.047458 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:24.047794 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:24.047798 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:24.047801 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:24.048054 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:35:24.048157 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 16:35:24.048738 (+ 581us) server_negotiation.cc:299] Negotiation successful
0925 16:35:24.048878 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 16:35:24.050887 13654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:24.036944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41289 (local address 127.0.98.3:58895)
0925 16:35:24.037390 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:24.037439 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 16:35:24.038304 (+ 865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:35:24.038859 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:35:24.038866 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:35:24.039121 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:35:24.039577 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:24.039589 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:24.040394 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:24.040397 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:24.041585 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:24.041596 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:24.042244 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:24.042246 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:24.042384 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:24.043062 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:35:24.043083 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:35:24.044815 (+ 1732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:35:24.046988 (+ 2173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:35:24.047001 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:35:24.047013 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:35:24.047253 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:35:24.047556 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:35:24.047559 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:35:24.047561 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:35:24.047699 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:35:24.048155 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:35:24.048162 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:35:24.048452 (+ 290us) client_negotiation.cc:769] Sending connection context
0925 16:35:24.049485 (+ 1033us) client_negotiation.cc:240] Negotiation successful
0925 16:35:24.049831 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":133,"threads_started":1}
I20260925 16:35:24.052326 13652 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41289
I20260925 16:35:24.052585 13652 heartbeater.cc:464] Registering TS with master...
I20260925 16:35:24.053146 13652 heartbeater.cc:511] Master 127.0.98.62:41289 requested a full tablet report, sending...
W20260925 16:35:24.054680 13652 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41289 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 16:35:24.054821 13652 heartbeater.cc:464] Registering TS with master...
I20260925 16:35:24.054998 13652 heartbeater.cc:511] Master 127.0.98.62:41289 requested a full tablet report, sending...
I20260925 16:35:24.055514 13655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:24.043132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43476 (local address 127.0.98.62:41289)
0925 16:35:24.044865 (+ 1733us) server_negotiation.cc:206] Beginning negotiation
0925 16:35:24.044874 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 16:35:24.044887 (+ 13us) server_negotiation.cc:407] Connection header received
0925 16:35:24.044923 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:35:24.044926 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:35:24.044985 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:35:24.045067 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:35:24.046732 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:24.047523 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:24.048763 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:24.049125 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:24.049228 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:24.050947 (+ 1719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:35:24.050950 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:35:24.050967 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:35:24.050993 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:35:24.052721 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:24.053215 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:24.053218 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:24.053220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:24.053304 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:24.053680 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:24.053683 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:24.053685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:24.053890 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:35:24.054092 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0925 16:35:24.055202 (+ 1110us) server_negotiation.cc:299] Negotiation successful
0925 16:35:24.055344 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":102,"threads_started":1}
I20260925 16:35:24.056103 13652 heartbeater.cc:464] Registering TS with master...
I20260925 16:35:24.056278 13652 heartbeater.cc:511] Master 127.0.98.62:41289 requested a full tablet report, sending...
W20260925 16:35:24.057011 13652 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 25 16:35:24 dist-test-slave-65pf krb5kdc[12190](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790354124, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:35:24.587239 13384 heartbeater.cc:464] Registering TS with master...
I20260925 16:35:24.587538 13384 heartbeater.cc:511] Master 127.0.98.62:41289 requested a full tablet report, sending...
I20260925 16:35:24.588625 13186 ts_manager.cc:194] Registered new tserver with Master: 05fc8758b3a44eb3b6f8d24fe6e7057f (127.0.98.1:36347)
I20260925 16:35:24.590535 13186 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58295
I20260925 16:35:24.812191 13518 heartbeater.cc:464] Registering TS with master...
I20260925 16:35:24.812485 13518 heartbeater.cc:511] Master 127.0.98.62:41289 requested a full tablet report, sending...
I20260925 16:35:24.813251 13186 ts_manager.cc:194] Registered new tserver with Master: 6771b685a2284b5b9eced07cb208d017 (127.0.98.2:35017)
I20260925 16:35:24.813993 13186 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59969
Sep 25 16:35:24 dist-test-slave-65pf krb5kdc[12190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354124, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 16:35:25.057250 13652 heartbeater.cc:464] Registering TS with master...
I20260925 16:35:25.057562 13652 heartbeater.cc:511] Master 127.0.98.62:41289 requested a full tablet report, sending...
I20260925 16:35:25.058508 13186 ts_manager.cc:194] Registered new tserver with Master: 7e2574024e5242cc9484c04cdec3a2e7 (127.0.98.3:46115)
I20260925 16:35:25.059388 13186 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58895
I20260925 16:35:25.059475 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 16:35:25.071491 13675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:25.061777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43486 (local address 127.0.98.62:41289)
0925 16:35:25.062036 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0925 16:35:25.062041 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:35:25.062055 (+ 14us) server_negotiation.cc:407] Connection header received
0925 16:35:25.062097 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:35:25.062099 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:35:25.062145 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:35:25.062239 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:35:25.063273 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:25.064115 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:25.065302 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:25.065671 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:25.065789 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:25.067385 (+ 1596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:35:25.067389 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:35:25.067392 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:35:25.067419 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:35:25.069256 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:25.069782 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:25.069785 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:25.069787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:25.069880 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:25.070172 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:25.070176 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:25.070178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:25.070370 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:35:25.070516 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 16:35:25.071166 (+ 650us) server_negotiation.cc:299] Negotiation successful
0925 16:35:25.071303 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":94,"threads_started":1}
I20260925 16:35:25.076306 13186 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43486:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
Sep 25 16:35:25 dist-test-slave-65pf krb5kdc[12190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354124, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260925 16:35:25.081431 13186 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:35:25.111797 13685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:25.100869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36347 (local address 127.0.0.1:37442)
0925 16:35:25.103758 (+ 2889us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:25.103812 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 16:35:25.103941 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:35:25.105992 (+ 2051us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:35:25.105998 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:35:25.106021 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:35:25.106159 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:25.106166 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:25.107620 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:25.107623 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:25.109775 (+ 2152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:25.109786 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:25.111192 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:25.111199 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:25.111321 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:25.111363 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 16:35:25.111477 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 16:35:25.111610 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2714,"thread_start_us":113,"threads_started":1}
I20260925 16:35:25.113731 13684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:25.102206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37442 (local address 127.0.98.1:36347)
0925 16:35:25.104156 (+ 1950us) server_negotiation.cc:206] Beginning negotiation
0925 16:35:25.104162 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:35:25.104186 (+ 24us) server_negotiation.cc:407] Connection header received
0925 16:35:25.104261 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:35:25.104266 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:35:25.104501 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:35:25.104649 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:35:25.106343 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:25.107488 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:25.109918 (+ 2430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:25.110819 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:25.110933 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:25.111115 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0925 16:35:25.113466 (+ 2351us) server_negotiation.cc:299] Negotiation successful
0925 16:35:25.113575 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1175,"thread_start_us":66,"threads_started":1}
I20260925 16:35:25.116417 13681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:25.100869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35017 (local address 127.0.0.1:48932)
0925 16:35:25.110749 (+ 9880us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:25.110764 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:35:25.110861 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:35:25.112287 (+ 1426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:35:25.112290 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:35:25.112306 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:35:25.112397 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:25.112402 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:25.113761 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:25.113764 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:25.114863 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:25.114873 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:25.116032 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:25.116036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:25.116118 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:25.116140 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 16:35:25.116203 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 16:35:25.116265 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9811,"mutex_wait_us":1528,"thread_start_us":86,"threads_started":1}
I20260925 16:35:25.116895 13683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:25.101849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48932 (local address 127.0.98.2:35017)
0925 16:35:25.103300 (+ 1451us) server_negotiation.cc:206] Beginning negotiation
0925 16:35:25.103308 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 16:35:25.110962 (+ 7654us) server_negotiation.cc:407] Connection header received
0925 16:35:25.111048 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:35:25.111054 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:35:25.111273 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:35:25.111435 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:35:25.112515 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:25.113641 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:25.114996 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:25.115750 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:25.115844 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:25.115975 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 16:35:25.116678 (+ 703us) server_negotiation.cc:299] Negotiation successful
0925 16:35:25.116769 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1328,"thread_start_us":114,"threads_started":1}
I20260925 16:35:25.117115 13300 tablet_service.cc:1467] Processing CreateTablet for tablet d006e2053c5e40d39698a18ddc6803a4 (DEFAULT_TABLE table=test-table [id=17f5cdc700f7462b920d731845abae39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:35:25.117666 13680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:25.101317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46115 (local address 127.0.0.1:57162)
0925 16:35:25.105748 (+ 4431us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:25.105764 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:35:25.105857 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:35:25.112785 (+ 6928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:35:25.112788 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:35:25.112804 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:35:25.112934 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:25.112940 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:25.115022 (+ 2082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:25.115025 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:25.116086 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:25.116097 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:25.117285 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:25.117289 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:25.117374 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:25.117394 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 16:35:25.117474 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 16:35:25.117536 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4362,"thread_start_us":93,"threads_started":1}
I20260925 16:35:25.118060 13682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:25.101602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57162 (local address 127.0.98.3:46115)
0925 16:35:25.111735 (+ 10133us) server_negotiation.cc:206] Beginning negotiation
0925 16:35:25.111742 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 16:35:25.111764 (+ 22us) server_negotiation.cc:407] Connection header received
0925 16:35:25.111835 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:35:25.111840 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:35:25.112022 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:35:25.112156 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:35:25.113069 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:25.114885 (+ 1816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:25.116228 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:25.116988 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:25.117089 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:25.117223 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 16:35:25.117826 (+ 603us) server_negotiation.cc:299] Negotiation successful
0925 16:35:25.117927 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10027,"thread_start_us":107,"threads_started":1}
I20260925 16:35:25.118275 13300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d006e2053c5e40d39698a18ddc6803a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:35:25.120887 13581 tablet_service.cc:1467] Processing CreateTablet for tablet d006e2053c5e40d39698a18ddc6803a4 (DEFAULT_TABLE table=test-table [id=17f5cdc700f7462b920d731845abae39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:35:25.121918 13581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d006e2053c5e40d39698a18ddc6803a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:35:25.122712 13438 tablet_service.cc:1467] Processing CreateTablet for tablet d006e2053c5e40d39698a18ddc6803a4 (DEFAULT_TABLE table=test-table [id=17f5cdc700f7462b920d731845abae39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:35:25.123646 13438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d006e2053c5e40d39698a18ddc6803a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:35:25.131927 13686 tablet_bootstrap.cc:492] T d006e2053c5e40d39698a18ddc6803a4 P 7e2574024e5242cc9484c04cdec3a2e7: Bootstrap starting.
I20260925 16:35:25.134256 13686 tablet_bootstrap.cc:654] T d006e2053c5e40d39698a18ddc6803a4 P 7e2574024e5242cc9484c04cdec3a2e7: Neither blocks nor log segments found. Creating new log.
I20260925 16:35:25.135195 13686 log.cc:824] T d006e2053c5e40d39698a18ddc6803a4 P 7e2574024e5242cc9484c04cdec3a2e7: Log is configured to *not* fsync() on all Append() calls
I20260925 16:35:25.137328 13686 tablet_bootstrap.cc:492] T d006e2053c5e40d39698a18ddc6803a4 P 7e2574024e5242cc9484c04cdec3a2e7: No bootstrap required, opened a new log
I20260925 16:35:25.137524 13686 ts_tablet_manager.cc:1397] T d006e2053c5e40d39698a18ddc6803a4 P 7e2574024e5242cc9484c04cdec3a2e7: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 16:35:25.141827 13686 raft_consensus.cc:348] T d006e2053c5e40d39698a18ddc6803a4 P 7e2574024e5242cc9484c04cdec3a2e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fc8758b3a44eb3b6f8d24fe6e7057f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36347 } } peers { permanent_uuid: "6771b685a2284b5b9eced07cb208d017" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35017 } } peers { permanent_uuid: "7e2574024e5242cc9484c04cdec3a2e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46115 } }
I20260925 16:35:25.142223 13686 raft_consensus.cc:374] T d006e2053c5e40d39698a18ddc6803a4 P 7e2574024e5242cc9484c04cdec3a2e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:35:25.142318 13686 raft_consensus.cc:729] T d006e2053c5e40d39698a18ddc6803a4 P 7e2574024e5242cc9484c04cdec3a2e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e2574024e5242cc9484c04cdec3a2e7, State: Initialized, Role: FOLLOWER
I20260925 16:35:25.142935 13688 tablet_bootstrap.cc:492] T d006e2053c5e40d39698a18ddc6803a4 P 6771b685a2284b5b9eced07cb208d017: Bootstrap starting.
I20260925 16:35:25.142903 13686 consensus_queue.cc:260] T d006e2053c5e40d39698a18ddc6803a4 P 7e2574024e5242cc9484c04cdec3a2e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fc8758b3a44eb3b6f8d24fe6e7057f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36347 } } peers { permanent_uuid: "6771b685a2284b5b9eced07cb208d017" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35017 } } peers { permanent_uuid: "7e2574024e5242cc9484c04cdec3a2e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46115 } }
I20260925 16:35:25.143946 13686 ts_tablet_manager.cc:1428] T d006e2053c5e40d39698a18ddc6803a4 P 7e2574024e5242cc9484c04cdec3a2e7: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260925 16:35:25.144333 13652 heartbeater.cc:503] Master 127.0.98.62:41289 was elected leader, sending a full tablet report...
I20260925 16:35:25.145366 13688 tablet_bootstrap.cc:654] T d006e2053c5e40d39698a18ddc6803a4 P 6771b685a2284b5b9eced07cb208d017: Neither blocks nor log segments found. Creating new log.
I20260925 16:35:25.145794 13690 tablet_bootstrap.cc:492] T d006e2053c5e40d39698a18ddc6803a4 P 05fc8758b3a44eb3b6f8d24fe6e7057f: Bootstrap starting.
I20260925 16:35:25.148573 13690 tablet_bootstrap.cc:654] T d006e2053c5e40d39698a18ddc6803a4 P 05fc8758b3a44eb3b6f8d24fe6e7057f: Neither blocks nor log segments found. Creating new log.
I20260925 16:35:25.149442 13690 log.cc:824] T d006e2053c5e40d39698a18ddc6803a4 P 05fc8758b3a44eb3b6f8d24fe6e7057f: Log is configured to *not* fsync() on all Append() calls
I20260925 16:35:25.150727 13688 log.cc:824] T d006e2053c5e40d39698a18ddc6803a4 P 6771b685a2284b5b9eced07cb208d017: Log is configured to *not* fsync() on all Append() calls
I20260925 16:35:25.153314 13690 tablet_bootstrap.cc:492] T d006e2053c5e40d39698a18ddc6803a4 P 05fc8758b3a44eb3b6f8d24fe6e7057f: No bootstrap required, opened a new log
I20260925 16:35:25.153525 13690 ts_tablet_manager.cc:1397] T d006e2053c5e40d39698a18ddc6803a4 P 05fc8758b3a44eb3b6f8d24fe6e7057f: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.004s
I20260925 16:35:25.157896 13690 raft_consensus.cc:348] T d006e2053c5e40d39698a18ddc6803a4 P 05fc8758b3a44eb3b6f8d24fe6e7057f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fc8758b3a44eb3b6f8d24fe6e7057f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36347 } } peers { permanent_uuid: "6771b685a2284b5b9eced07cb208d017" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35017 } } peers { permanent_uuid: "7e2574024e5242cc9484c04cdec3a2e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46115 } }
I20260925 16:35:25.158296 13690 raft_consensus.cc:374] T d006e2053c5e40d39698a18ddc6803a4 P 05fc8758b3a44eb3b6f8d24fe6e7057f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:35:25.158406 13690 raft_consensus.cc:729] T d006e2053c5e40d39698a18ddc6803a4 P 05fc8758b3a44eb3b6f8d24fe6e7057f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05fc8758b3a44eb3b6f8d24fe6e7057f, State: Initialized, Role: FOLLOWER
I20260925 16:35:25.158987 13690 consensus_queue.cc:260] T d006e2053c5e40d39698a18ddc6803a4 P 05fc8758b3a44eb3b6f8d24fe6e7057f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fc8758b3a44eb3b6f8d24fe6e7057f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36347 } } peers { permanent_uuid: "6771b685a2284b5b9eced07cb208d017" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35017 } } peers { permanent_uuid: "7e2574024e5242cc9484c04cdec3a2e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46115 } }
I20260925 16:35:25.161736 13690 ts_tablet_manager.cc:1428] T d006e2053c5e40d39698a18ddc6803a4 P 05fc8758b3a44eb3b6f8d24fe6e7057f: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260925 16:35:25.161963 13688 tablet_bootstrap.cc:492] T d006e2053c5e40d39698a18ddc6803a4 P 6771b685a2284b5b9eced07cb208d017: No bootstrap required, opened a new log
I20260925 16:35:25.162128 13384 heartbeater.cc:503] Master 127.0.98.62:41289 was elected leader, sending a full tablet report...
I20260925 16:35:25.162210 13688 ts_tablet_manager.cc:1397] T d006e2053c5e40d39698a18ddc6803a4 P 6771b685a2284b5b9eced07cb208d017: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.005s
I20260925 16:35:25.166556 13688 raft_consensus.cc:348] T d006e2053c5e40d39698a18ddc6803a4 P 6771b685a2284b5b9eced07cb208d017 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fc8758b3a44eb3b6f8d24fe6e7057f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36347 } } peers { permanent_uuid: "6771b685a2284b5b9eced07cb208d017" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35017 } } peers { permanent_uuid: "7e2574024e5242cc9484c04cdec3a2e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46115 } }
I20260925 16:35:25.166914 13688 raft_consensus.cc:374] T d006e2053c5e40d39698a18ddc6803a4 P 6771b685a2284b5b9eced07cb208d017 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:35:25.167008 13688 raft_consensus.cc:729] T d006e2053c5e40d39698a18ddc6803a4 P 6771b685a2284b5b9eced07cb208d017 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6771b685a2284b5b9eced07cb208d017, State: Initialized, Role: FOLLOWER
I20260925 16:35:25.167564 13688 consensus_queue.cc:260] T d006e2053c5e40d39698a18ddc6803a4 P 6771b685a2284b5b9eced07cb208d017 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fc8758b3a44eb3b6f8d24fe6e7057f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36347 } } peers { permanent_uuid: "6771b685a2284b5b9eced07cb208d017" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35017 } } peers { permanent_uuid: "7e2574024e5242cc9484c04cdec3a2e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46115 } }
I20260925 16:35:25.168607 13688 ts_tablet_manager.cc:1428] T d006e2053c5e40d39698a18ddc6803a4 P 6771b685a2284b5b9eced07cb208d017: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260925 16:35:25.168988 13518 heartbeater.cc:503] Master 127.0.98.62:41289 was elected leader, sending a full tablet report...
I20260925 16:35:25.179184 13693 raft_consensus.cc:482] T d006e2053c5e40d39698a18ddc6803a4 P 05fc8758b3a44eb3b6f8d24fe6e7057f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:35:25.179397 13693 raft_consensus.cc:504] T d006e2053c5e40d39698a18ddc6803a4 P 05fc8758b3a44eb3b6f8d24fe6e7057f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fc8758b3a44eb3b6f8d24fe6e7057f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36347 } } peers { permanent_uuid: "6771b685a2284b5b9eced07cb208d017" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35017 } } peers { permanent_uuid: "7e2574024e5242cc9484c04cdec3a2e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46115 } }
I20260925 16:35:25.180732 13693 leader_election.cc:288] T d006e2053c5e40d39698a18ddc6803a4 P 05fc8758b3a44eb3b6f8d24fe6e7057f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6771b685a2284b5b9eced07cb208d017 (127.0.98.2:35017), 7e2574024e5242cc9484c04cdec3a2e7 (127.0.98.3:46115)
I20260925 16:35:25.186059 13683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:25.181318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58415 (local address 127.0.98.2:35017)
0925 16:35:25.181437 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0925 16:35:25.181442 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:35:25.181727 (+ 285us) server_negotiation.cc:407] Connection header received
0925 16:35:25.181917 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:35:25.181920 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:35:25.181965 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:35:25.182011 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:35:25.182402 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:25.183202 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:25.184712 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:25.185358 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:25.185459 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:25.185534 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 16:35:25.185844 (+ 310us) server_negotiation.cc:299] Negotiation successful
0925 16:35:25.185919 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 16:35:25.186425 13695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:25.180942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35017 (local address 127.0.98.1:58415)
0925 16:35:25.181641 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:25.181656 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:35:25.181801 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:35:25.182134 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:35:25.182137 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:35:25.182152 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:35:25.182283 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:25.182289 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:25.183333 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:25.183336 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:25.184577 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:25.184587 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:25.185595 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:25.185599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:25.185705 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:25.185729 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 16:35:25.186191 (+ 462us) client_negotiation.cc:240] Negotiation successful
0925 16:35:25.186268 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":124,"threads_started":1}
I20260925 16:35:25.187146 13472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d006e2053c5e40d39698a18ddc6803a4" candidate_uuid: "05fc8758b3a44eb3b6f8d24fe6e7057f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6771b685a2284b5b9eced07cb208d017" is_pre_election: true
I20260925 16:35:25.187448 13472 raft_consensus.cc:2454] T d006e2053c5e40d39698a18ddc6803a4 P 6771b685a2284b5b9eced07cb208d017 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05fc8758b3a44eb3b6f8d24fe6e7057f in term 0.
I20260925 16:35:25.187945 13272 leader_election.cc:302] T d006e2053c5e40d39698a18ddc6803a4 P 05fc8758b3a44eb3b6f8d24fe6e7057f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05fc8758b3a44eb3b6f8d24fe6e7057f, 6771b685a2284b5b9eced07cb208d017; no voters:
I20260925 16:35:25.188292 13693 raft_consensus.cc:2781] T d006e2053c5e40d39698a18ddc6803a4 P 05fc8758b3a44eb3b6f8d24fe6e7057f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:35:25.188367 13693 raft_consensus.cc:482] T d006e2053c5e40d39698a18ddc6803a4 P 05fc8758b3a44eb3b6f8d24fe6e7057f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:35:25.188405 13693 raft_consensus.cc:3037] T d006e2053c5e40d39698a18ddc6803a4 P 05fc8758b3a44eb3b6f8d24fe6e7057f [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:35:25.189360 13682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:25.181857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38205 (local address 127.0.98.3:46115)
0925 16:35:25.181981 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 16:35:25.181985 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:35:25.184862 (+ 2877us) server_negotiation.cc:407] Connection header received
0925 16:35:25.185029 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:35:25.185031 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:35:25.185097 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:35:25.185140 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:35:25.185503 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:25.186234 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:25.188136 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:25.188700 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:25.188775 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:25.188837 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 16:35:25.189176 (+ 339us) server_negotiation.cc:299] Negotiation successful
0925 16:35:25.189243 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 16:35:25.189749 13696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:25.181499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46115 (local address 127.0.98.1:38205)
0925 16:35:25.184791 (+ 3292us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:25.184803 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 16:35:25.184925 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:35:25.185253 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:35:25.185256 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:35:25.185270 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:35:25.185387 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:25.185392 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:25.186787 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:25.186790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:25.188008 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:25.188018 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:25.188964 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:25.188967 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:25.189060 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:25.189078 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 16:35:25.189524 (+ 446us) client_negotiation.cc:240] Negotiation successful
0925 16:35:25.189589 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3233,"thread_start_us":96,"threads_started":1}
I20260925 16:35:25.189731 13693 raft_consensus.cc:504] T d006e2053c5e40d39698a18ddc6803a4 P 05fc8758b3a44eb3b6f8d24fe6e7057f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fc8758b3a44eb3b6f8d24fe6e7057f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36347 } } peers { permanent_uuid: "6771b685a2284b5b9eced07cb208d017" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35017 } } peers { permanent_uuid: "7e2574024e5242cc9484c04cdec3a2e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46115 } }
I20260925 16:35:25.190071 13693 leader_election.cc:288] T d006e2053c5e40d39698a18ddc6803a4 P 05fc8758b3a44eb3b6f8d24fe6e7057f [CANDIDATE]: Term 1 election: Requested vote from peers 6771b685a2284b5b9eced07cb208d017 (127.0.98.2:35017), 7e2574024e5242cc9484c04cdec3a2e7 (127.0.98.3:46115)
I20260925 16:35:25.190251 13606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d006e2053c5e40d39698a18ddc6803a4" candidate_uuid: "05fc8758b3a44eb3b6f8d24fe6e7057f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e2574024e5242cc9484c04cdec3a2e7" is_pre_election: true
I20260925 16:35:25.190542 13606 raft_consensus.cc:2454] T d006e2053c5e40d39698a18ddc6803a4 P 7e2574024e5242cc9484c04cdec3a2e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05fc8758b3a44eb3b6f8d24fe6e7057f in term 0.
I20260925 16:35:25.190600 13472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d006e2053c5e40d39698a18ddc6803a4" candidate_uuid: "05fc8758b3a44eb3b6f8d24fe6e7057f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6771b685a2284b5b9eced07cb208d017"
I20260925 16:35:25.190725 13472 raft_consensus.cc:3037] T d006e2053c5e40d39698a18ddc6803a4 P 6771b685a2284b5b9eced07cb208d017 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:35:25.191613 13606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d006e2053c5e40d39698a18ddc6803a4" candidate_uuid: "05fc8758b3a44eb3b6f8d24fe6e7057f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e2574024e5242cc9484c04cdec3a2e7"
I20260925 16:35:25.191711 13606 raft_consensus.cc:3037] T d006e2053c5e40d39698a18ddc6803a4 P 7e2574024e5242cc9484c04cdec3a2e7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:35:25.191951 13472 raft_consensus.cc:2454] T d006e2053c5e40d39698a18ddc6803a4 P 6771b685a2284b5b9eced07cb208d017 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05fc8758b3a44eb3b6f8d24fe6e7057f in term 1.
I20260925 16:35:25.192247 13272 leader_election.cc:302] T d006e2053c5e40d39698a18ddc6803a4 P 05fc8758b3a44eb3b6f8d24fe6e7057f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05fc8758b3a44eb3b6f8d24fe6e7057f, 6771b685a2284b5b9eced07cb208d017; no voters:
I20260925 16:35:25.192397 13693 raft_consensus.cc:2781] T d006e2053c5e40d39698a18ddc6803a4 P 05fc8758b3a44eb3b6f8d24fe6e7057f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:35:25.192577 13693 raft_consensus.cc:686] T d006e2053c5e40d39698a18ddc6803a4 P 05fc8758b3a44eb3b6f8d24fe6e7057f [term 1 LEADER]: Becoming Leader. State: Replica: 05fc8758b3a44eb3b6f8d24fe6e7057f, State: Running, Role: LEADER
I20260925 16:35:25.192822 13606 raft_consensus.cc:2454] T d006e2053c5e40d39698a18ddc6803a4 P 7e2574024e5242cc9484c04cdec3a2e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05fc8758b3a44eb3b6f8d24fe6e7057f in term 1.
I20260925 16:35:25.192904 13693 consensus_queue.cc:237] T d006e2053c5e40d39698a18ddc6803a4 P 05fc8758b3a44eb3b6f8d24fe6e7057f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fc8758b3a44eb3b6f8d24fe6e7057f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36347 } } peers { permanent_uuid: "6771b685a2284b5b9eced07cb208d017" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35017 } } peers { permanent_uuid: "7e2574024e5242cc9484c04cdec3a2e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46115 } }
I20260925 16:35:25.196316 13186 catalog_manager.cc:5495] T d006e2053c5e40d39698a18ddc6803a4 P 05fc8758b3a44eb3b6f8d24fe6e7057f reported cstate change: term changed from 0 to 1, leader changed from <none> to 05fc8758b3a44eb3b6f8d24fe6e7057f (127.0.98.1). New cstate: current_term: 1 leader_uuid: "05fc8758b3a44eb3b6f8d24fe6e7057f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fc8758b3a44eb3b6f8d24fe6e7057f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36347 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6771b685a2284b5b9eced07cb208d017" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e2574024e5242cc9484c04cdec3a2e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46115 } health_report { overall_health: UNKNOWN } } }
I20260925 16:35:25.212138 13684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:25.207218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37458 (local address 127.0.98.1:36347)
0925 16:35:25.207335 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 16:35:25.207339 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:35:25.207413 (+ 74us) server_negotiation.cc:407] Connection header received
0925 16:35:25.207569 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:35:25.207572 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:35:25.207632 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:35:25.207738 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:35:25.208110 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:25.208835 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:25.210060 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:25.210398 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:25.210490 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:25.210895 (+ 405us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:35:25.211438 (+ 543us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:35:25.211547 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 16:35:25.211930 (+ 383us) server_negotiation.cc:299] Negotiation successful
0925 16:35:25.211987 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 16:35:25.229202 13606 raft_consensus.cc:1261] T d006e2053c5e40d39698a18ddc6803a4 P 7e2574024e5242cc9484c04cdec3a2e7 [term 1 FOLLOWER]: Refusing update from remote peer 05fc8758b3a44eb3b6f8d24fe6e7057f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:35:25.230120 13693 consensus_queue.cc:1035] T d006e2053c5e40d39698a18ddc6803a4 P 05fc8758b3a44eb3b6f8d24fe6e7057f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e2574024e5242cc9484c04cdec3a2e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:35:25.237181 13472 raft_consensus.cc:1261] T d006e2053c5e40d39698a18ddc6803a4 P 6771b685a2284b5b9eced07cb208d017 [term 1 FOLLOWER]: Refusing update from remote peer 05fc8758b3a44eb3b6f8d24fe6e7057f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:35:25.237972 13693 consensus_queue.cc:1035] T d006e2053c5e40d39698a18ddc6803a4 P 05fc8758b3a44eb3b6f8d24fe6e7057f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6771b685a2284b5b9eced07cb208d017" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:35:25.259177 13702 mvcc.cc:205] Tried to move back new op lower bound from 7333290496930484224 to 7333290496793108480. Current Snapshot: MvccSnapshot[applied={T|T < 7333290496930484224}]
I20260925 16:35:25.263652 13701 mvcc.cc:205] Tried to move back new op lower bound from 7333290496930484224 to 7333290496793108480. Current Snapshot: MvccSnapshot[applied={T|T < 7333290496930484224}]
I20260925 16:35:25.280432 13185 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43486:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:35:25.280649 13185 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43486:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:35:25.283684 13185 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b459797117b049e3b92eb580851e8621: Sending DeleteTablet for 3 replicas of tablet d006e2053c5e40d39698a18ddc6803a4
I20260925 16:35:25.284699 13438 tablet_service.cc:1514] Processing DeleteTablet for tablet d006e2053c5e40d39698a18ddc6803a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:35:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48932
I20260925 16:35:25.284857 13581 tablet_service.cc:1514] Processing DeleteTablet for tablet d006e2053c5e40d39698a18ddc6803a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:35:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57162
I20260925 16:35:25.285737 13300 tablet_service.cc:1514] Processing DeleteTablet for tablet d006e2053c5e40d39698a18ddc6803a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:35:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37442
I20260925 16:35:25.287912 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 13224
I20260925 16:35:25.292871 13711 tablet_replica.cc:330] T d006e2053c5e40d39698a18ddc6803a4 P 7e2574024e5242cc9484c04cdec3a2e7: stopping tablet replica
I20260925 16:35:25.293295 13711 raft_consensus.cc:2229] T d006e2053c5e40d39698a18ddc6803a4 P 7e2574024e5242cc9484c04cdec3a2e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:35:25.293774 13711 raft_consensus.cc:2258] T d006e2053c5e40d39698a18ddc6803a4 P 7e2574024e5242cc9484c04cdec3a2e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:35:25.295475 13711 ts_tablet_manager.cc:1905] T d006e2053c5e40d39698a18ddc6803a4 P 7e2574024e5242cc9484c04cdec3a2e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:35:25.302812 13710 tablet_replica.cc:330] T d006e2053c5e40d39698a18ddc6803a4 P 6771b685a2284b5b9eced07cb208d017: stopping tablet replica
I20260925 16:35:25.303261 13710 raft_consensus.cc:2229] T d006e2053c5e40d39698a18ddc6803a4 P 6771b685a2284b5b9eced07cb208d017 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:35:25.303675 13710 raft_consensus.cc:2258] T d006e2053c5e40d39698a18ddc6803a4 P 6771b685a2284b5b9eced07cb208d017 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:35:25.304137 13711 ts_tablet_manager.cc:1918] T d006e2053c5e40d39698a18ddc6803a4 P 7e2574024e5242cc9484c04cdec3a2e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:35:25.304893 13711 log.cc:1196] T d006e2053c5e40d39698a18ddc6803a4 P 7e2574024e5242cc9484c04cdec3a2e7: Deleting WAL directory at /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-2/wal/wals/d006e2053c5e40d39698a18ddc6803a4
I20260925 16:35:25.305955 13711 ts_tablet_manager.cc:1939] T d006e2053c5e40d39698a18ddc6803a4 P 7e2574024e5242cc9484c04cdec3a2e7: Deleting consensus metadata
I20260925 16:35:25.306937 13710 ts_tablet_manager.cc:1905] T d006e2053c5e40d39698a18ddc6803a4 P 6771b685a2284b5b9eced07cb208d017: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:35:25.309100 13173 catalog_manager.cc:4841] TS 7e2574024e5242cc9484c04cdec3a2e7 (127.0.98.3:46115): tablet d006e2053c5e40d39698a18ddc6803a4 (table test-table [id=17f5cdc700f7462b920d731845abae39]) successfully deleted
I20260925 16:35:25.311882 13710 ts_tablet_manager.cc:1918] T d006e2053c5e40d39698a18ddc6803a4 P 6771b685a2284b5b9eced07cb208d017: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:35:25.312034 13710 log.cc:1196] T d006e2053c5e40d39698a18ddc6803a4 P 6771b685a2284b5b9eced07cb208d017: Deleting WAL directory at /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353966814071-392-0/minicluster-data/ts-1/wal/wals/d006e2053c5e40d39698a18ddc6803a4
I20260925 16:35:25.312407 13710 ts_tablet_manager.cc:1939] T d006e2053c5e40d39698a18ddc6803a4 P 6771b685a2284b5b9eced07cb208d017: Deleting consensus metadata
W20260925 16:35:25.313138 13172 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36347 (error 108)
I20260925 16:35:25.315145 13171 catalog_manager.cc:4841] TS 6771b685a2284b5b9eced07cb208d017 (127.0.98.2:35017): tablet d006e2053c5e40d39698a18ddc6803a4 (table test-table [id=17f5cdc700f7462b920d731845abae39]) successfully deleted
I20260925 16:35:25.315543 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 13386
I20260925 16:35:25.323005 13680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:25.322398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36347 (local address 127.0.0.1:37470)
0925 16:35:25.322816 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:25.322901 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
W20260925 16:35:25.323323 13172 catalog_manager.cc:4584] TS 05fc8758b3a44eb3b6f8d24fe6e7057f (127.0.98.1:36347): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d006e2053c5e40d39698a18ddc6803a4: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36347: connect: Connection refused (error 111)
I20260925 16:35:25.327003 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 13521
I20260925 16:35:25.334496 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 13153
I20260925 16:35:25.342413 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 16:35:25.677381 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 16:35:25.677538 392 mini_ranger.cc:66] Stopping Ranger...
I20260925 16:35:26.022702 392 mini_ranger.cc:68] Stopped Ranger
2026-09-25 16:35:26.022 UTC [12220] LOG: received smart shutdown request
2026-09-25 16:35:26.027 UTC [12220] LOG: background worker "logical replication launcher" (PID 12229) exited with exit code 1
2026-09-25 16:35:26.027 UTC [12224] LOG: shutting down
2026-09-25 16:35:26.189 UTC [12220] LOG: database system is shut down
2026-09-25T16:35:26Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48436 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353966814071-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:35:26 dist-test-slave-65pf krb5kdc[13722](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:35:26 dist-test-slave-65pf krb5kdc[13722](info): set up 2 sockets
Sep 25 16:35:26 dist-test-slave-65pf krb5kdc[13722](info): commencing operation
krb5kdc: starting...
W20260925 16:35:27.350750 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353966814071-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353966814071-392-0/krb5kdc/test-user.keytab.
Sep 25 16:35:27 dist-test-slave-65pf krb5kdc[13722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:35:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:35:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 16:35:27.469425 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353966814071-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353966814071-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353966814071-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40127
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40127,127.0.98.61:44969,127.0.98.60:45807
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:35:27.576023 13735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:35:27.576278 13735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:35:27.576318 13735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:35:27.580363 13735 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:35:27.580420 13735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:35:27.580451 13735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:35:27.580469 13735 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:35:27.580529 13735 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:35:27.585628 13735 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353966814071-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353966814071-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40127,127.0.98.61:44969,127.0.98.60:45807
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353966814071-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:35:27.586679 13735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:35:27.587695 13735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:35:27.596053 13740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:35:27.596087 13741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:35:27.596088 13743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:35:27.596364 13735 server_base.cc:1034] running on GCE node
I20260925 16:35:27.596953 13735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:35:27.597996 13735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:35:27.599179 13735 hybrid_clock.cc:648] HybridClock initialized: now 1790354127599152 us; error 34 us; skew 500 ppm
Sep 25 16:35:27 dist-test-slave-65pf krb5kdc[13722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354127, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:35:27.601903 13735 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 16:35:27.602877 13735 webserver.cc:460] Webserver started at http://127.0.98.62:37103/ using document root <none> and password file <none>
I20260925 16:35:27.603426 13735 fs_manager.cc:359] Metadata directory not provided
I20260925 16:35:27.603520 13735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:35:27.603746 13735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:35:27.606122 13735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353966814071-392-0/minicluster-data/master-0/data/instance:
uuid: "34a62985747e49a7beeb56ba4727006b"
format_stamp: "Formatted at 2026-09-25 16:35:27 on dist-test-slave-65pf"
server_key: "8dfe12efc937a9cd49981177df7e79e1"
server_key_iv: "8c81d641912a56dedf20f45ef052be9a"
server_key_version: "encryptionkey@0"
I20260925 16:35:27.606829 13735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353966814071-392-0/minicluster-data/master-0/wal/instance:
uuid: "34a62985747e49a7beeb56ba4727006b"
format_stamp: "Formatted at 2026-09-25 16:35:27 on dist-test-slave-65pf"
server_key: "8dfe12efc937a9cd49981177df7e79e1"
server_key_iv: "8c81d641912a56dedf20f45ef052be9a"
server_key_version: "encryptionkey@0"
I20260925 16:35:27.610337 13735 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 16:35:27.612709 13750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:35:27.613909 13735 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:35:27.614042 13735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353966814071-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353966814071-392-0/minicluster-data/master-0/wal
uuid: "34a62985747e49a7beeb56ba4727006b"
format_stamp: "Formatted at 2026-09-25 16:35:27 on dist-test-slave-65pf"
server_key: "8dfe12efc937a9cd49981177df7e79e1"
server_key_iv: "8c81d641912a56dedf20f45ef052be9a"
server_key_version: "encryptionkey@0"
I20260925 16:35:27.614145 13735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353966814071-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353966814071-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353966814071-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:35:27.629045 13735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:35:27.631853 13735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:35:27.632057 13735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:35:27.638835 13735 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40127
I20260925 16:35:27.638907 13802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40127 every 8 connection(s)
I20260925 16:35:27.639847 13735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:35:27.642417 13803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:35:27.645058 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 13735
I20260925 16:35:27.645206 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353966814071-392-0/minicluster-data/master-0/wal/instance
I20260925 16:35:27.645555 392 external_mini_cluster.cc:1428] Setting key a7d438c5e31d83e763b23b5df55453cb
I20260925 16:35:27.646724 13803 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40127 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 25 16:35:27 dist-test-slave-65pf krb5kdc[13722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354127, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:35:27.662514 13806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:27.649006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40127 (local address 127.0.0.1:59538)
0925 16:35:27.649440 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:27.649499 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 16:35:27.650372 (+ 873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:35:27.651199 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:35:27.651203 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:35:27.651541 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:35:27.651806 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:27.651819 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:27.653533 (+ 1714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:27.653535 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:27.654327 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:27.654336 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:27.655112 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:27.655115 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:27.655191 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:27.655760 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:35:27.655775 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:35:27.657335 (+ 1560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:35:27.659601 (+ 2266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:35:27.659611 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:35:27.659620 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:35:27.659828 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:35:27.660143 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:35:27.660145 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:35:27.660147 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:35:27.660253 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:35:27.660813 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:35:27.660819 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:35:27.660950 (+ 131us) client_negotiation.cc:769] Sending connection context
0925 16:35:27.661114 (+ 164us) client_negotiation.cc:240] Negotiation successful
0925 16:35:27.661567 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":130,"thread_start_us":123,"threads_started":1}
I20260925 16:35:27.662518 13807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:27.649822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59538 (local address 127.0.98.62:40127)
0925 16:35:27.650109 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0925 16:35:27.650122 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 16:35:27.650172 (+ 50us) server_negotiation.cc:407] Connection header received
0925 16:35:27.650728 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:35:27.650736 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:35:27.650993 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:35:27.651211 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:35:27.651960 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:27.653411 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:27.654437 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:27.654973 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:27.655090 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:27.657487 (+ 2397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:35:27.657492 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:35:27.657501 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:35:27.657520 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:35:27.659504 (+ 1984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:27.659940 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:27.659945 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:27.659947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:27.660048 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:27.660344 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:27.660347 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:27.660349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:27.660699 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:35:27.660828 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 16:35:27.661185 (+ 357us) server_negotiation.cc:299] Negotiation successful
0925 16:35:27.661516 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":180,"thread_start_us":89,"threads_started":1}
I20260925 16:35:27.665174 13803 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44969 } has no permanent_uuid. Determining permanent_uuid...
I20260925 16:35:27.665850 13806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:27.665619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44969 (local address 127.0.0.1:57762)
0925 16:35:27.665727 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:27.665790 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 16:35:27.666054 13751 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44969: connect: Connection refused (error 111)
I20260925 16:35:27.666885 13806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:27.666686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44969 (local address 127.0.0.1:57774)
0925 16:35:27.666784 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:27.666834 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 16:35:27.667042 13803 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:44969: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44969: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353966814071-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353966814071-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260925 16:35:27.671630 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353966814071-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353966814071-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353966814071-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353966814071-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:44969
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353966814071-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40127,127.0.98.61:44969,127.0.98.60:45807
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 16:35:27.716290 13803 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44969 } attempt: 1
I20260925 16:35:27.717193 13806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:27.716849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44969 (local address 127.0.0.1:57790)
0925 16:35:27.717026 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:27.717101 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260925 16:35:27.717926 13806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:27.717705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44969 (local address 127.0.0.1:57794)
0925 16:35:27.717809 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:27.717861 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 16:35:27.718063 13803 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:44969: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44969: connect: Connection refused (error 111)
W20260925 16:35:27.779347 13810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:35:27.779594 13810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:35:27.779634 13810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:35:27.783631 13810 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:35:27.783684 13810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:35:27.783703 13810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:35:27.783720 13810 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:35:27.783769 13810 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:35:27.786247 13803 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44969 } attempt: 2
I20260925 16:35:27.787127 13806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:27.786800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44969 (local address 127.0.0.1:57800)
0925 16:35:27.786964 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:27.787035 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260925 16:35:27.787819 13806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:27.787608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44969 (local address 127.0.0.1:57816)
0925 16:35:27.787710 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:27.787759 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260925 16:35:27.787945 13803 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:44969: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44969: connect: Connection refused (error 111)
I20260925 16:35:27.789402 13810 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353966814071-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353966814071-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40127,127.0.98.61:44969,127.0.98.60:45807
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353966814071-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:44969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353966814071-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353966814071-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:35:27.790321 13810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:35:27.791304 13810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:35:27.799357 13816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:35:27.799366 13818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:35:27.799366 13815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:35:27.799582 13810 server_base.cc:1034] running on GCE node
I20260925 16:35:27.800258 13810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:35:27.801306 13810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:35:27.802460 13810 hybrid_clock.cc:648] HybridClock initialized: now 1790354127802441 us; error 30 us; skew 500 ppm
Sep 25 16:35:27 dist-test-slave-65pf krb5kdc[13722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354127, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:35:27.805205 13810 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260925 16:35:27.806099 13810 webserver.cc:460] Webserver started at http://127.0.98.61:34769/ using document root <none> and password file <none>
I20260925 16:35:27.806706 13810 fs_manager.cc:359] Metadata directory not provided
I20260925 16:35:27.806770 13810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:35:27.807004 13810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:35:27.808940 13810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353966814071-392-0/minicluster-data/master-1/data/instance:
uuid: "01d9bb013eee4cdbae694f9a28aa4da3"
format_stamp: "Formatted at 2026-09-25 16:35:27 on dist-test-slave-65pf"
server_key: "fe802e67a84022ba4d428bcddc386e65"
server_key_iv: "10ca4a1d5cd2265d65e9668e9e99bbee"
server_key_version: "encryptionkey@0"
I20260925 16:35:27.809489 13810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353966814071-392-0/minicluster-data/master-1/wal/instance:
uuid: "01d9bb013eee4cdbae694f9a28aa4da3"
format_stamp: "Formatted at 2026-09-25 16:35:27 on dist-test-slave-65pf"
server_key: "fe802e67a84022ba4d428bcddc386e65"
server_key_iv: "10ca4a1d5cd2265d65e9668e9e99bbee"
server_key_version: "encryptionkey@0"
I20260925 16:35:27.813036 13810 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 16:35:27.815290 13825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:35:27.816430 13810 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:35:27.816570 13810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353966814071-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353966814071-392-0/minicluster-data/master-1/wal
uuid: "01d9bb013eee4cdbae694f9a28aa4da3"
format_stamp: "Formatted at 2026-09-25 16:35:27 on dist-test-slave-65pf"
server_key: "fe802e67a84022ba4d428bcddc386e65"
server_key_iv: "10ca4a1d5cd2265d65e9668e9e99bbee"
server_key_version: "encryptionkey@0"
I20260925 16:35:27.816679 13810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353966814071-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353966814071-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353966814071-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:35:27.833789 13810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:35:27.837145 13810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:35:27.837318 13810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:35:27.844295 13810 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:44969
I20260925 16:35:27.844321 13877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:44969 every 8 connection(s)
I20260925 16:35:27.845199 13810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353966814071-392-0/minicluster-data/master-1/data/info.pb
I20260925 16:35:27.847393 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 13810
I20260925 16:35:27.847532 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353966814071-392-0/minicluster-data/master-1/wal/instance
I20260925 16:35:27.847815 392 external_mini_cluster.cc:1428] Setting key d4aa044d826a08906768a1e7f612444f
I20260925 16:35:27.848030 13878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:35:27.851658 13878 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40127 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 25 16:35:27 dist-test-slave-65pf krb5kdc[13722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354127, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:35:27.865193 13807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:27.853459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59550 (local address 127.0.98.62:40127)
0925 16:35:27.853643 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 16:35:27.853649 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:35:27.854380 (+ 731us) server_negotiation.cc:407] Connection header received
0925 16:35:27.855360 (+ 980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:35:27.855363 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:35:27.855412 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:35:27.855494 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:35:27.856678 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:27.857246 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:27.858433 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:27.858897 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:27.859033 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:27.861610 (+ 2577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:35:27.861613 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:35:27.861615 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:35:27.861638 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:35:27.863115 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:27.863641 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:27.863645 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:27.863646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:27.863718 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:27.864050 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:27.864054 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:27.864056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:27.864239 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:35:27.864348 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 16:35:27.864877 (+ 529us) server_negotiation.cc:299] Negotiation successful
0925 16:35:27.865016 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 16:35:27.866053 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:27.853771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40127 (local address 127.0.0.1:59550)
0925 16:35:27.854212 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:27.854261 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 16:35:27.855169 (+ 908us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:35:27.855659 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:35:27.855665 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:35:27.856061 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:35:27.856551 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:27.856564 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:27.857357 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:27.857360 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:27.858307 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:27.858316 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:27.859039 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:27.859042 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:27.859237 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:27.859853 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:35:27.859876 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:35:27.861428 (+ 1552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:35:27.863233 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:35:27.863240 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:35:27.863256 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:35:27.863520 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:35:27.863809 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:35:27.863813 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:35:27.863816 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:35:27.863953 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:35:27.864343 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:35:27.864349 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:35:27.864619 (+ 270us) client_negotiation.cc:769] Sending connection context
0925 16:35:27.864819 (+ 200us) client_negotiation.cc:240] Negotiation successful
0925 16:35:27.865141 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":135,"threads_started":1}
I20260925 16:35:27.867425 13878 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44969 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 16:35:27 dist-test-slave-65pf krb5kdc[13722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354127, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353966814071-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353966814071-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260925 16:35:27.873497 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353966814071-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353966814071-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353966814071-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353966814071-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45807
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353966814071-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40127,127.0.98.61:44969,127.0.98.60:45807
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 16:35:27.876435 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:27.867907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44969 (local address 127.0.0.1:57830)
0925 16:35:27.868021 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:27.868037 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:35:27.868110 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:35:27.868708 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:35:27.868711 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:35:27.868837 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:35:27.868922 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:27.868927 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:27.869877 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:27.869880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:27.870444 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:27.870452 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:27.871082 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:27.871084 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:27.871188 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:27.871654 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:35:27.871664 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:35:27.872913 (+ 1249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:35:27.874939 (+ 2026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:35:27.874946 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:35:27.874949 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:35:27.875218 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:35:27.875484 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:35:27.875487 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:35:27.875489 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:35:27.875595 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:35:27.876077 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:35:27.876079 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:35:27.876152 (+ 73us) client_negotiation.cc:769] Sending connection context
0925 16:35:27.876235 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 16:35:27.876326 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 16:35:27.876772 13883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:27.868064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57830 (local address 127.0.98.61:44969)
0925 16:35:27.868327 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0925 16:35:27.868335 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 16:35:27.868354 (+ 19us) server_negotiation.cc:407] Connection header received
0925 16:35:27.868388 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:35:27.868392 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:35:27.868593 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:35:27.868700 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:35:27.869050 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:27.869787 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:27.870559 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:27.870985 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:27.871106 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:27.873074 (+ 1968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:35:27.873081 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:35:27.873091 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:35:27.873111 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:35:27.874832 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:27.875319 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:27.875324 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:27.875325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:27.875394 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:27.875692 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:27.875694 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:27.875695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:27.875931 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:35:27.876078 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 16:35:27.876347 (+ 269us) server_negotiation.cc:299] Negotiation successful
0925 16:35:27.876504 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":97,"threads_started":1}
I20260925 16:35:27.878894 13878 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45807 } has no permanent_uuid. Determining permanent_uuid...
I20260925 16:35:27.879089 13803 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44969 } attempt: 3
I20260925 16:35:27.879680 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:27.879411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45807 (local address 127.0.0.1:37212)
0925 16:35:27.879532 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:27.879616 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 16:35:27.879896 13827 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45807: connect: Connection refused (error 111)
I20260925 16:35:27.880707 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:27.880495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45807 (local address 127.0.0.1:37226)
0925 16:35:27.880613 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:27.880661 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 16:35:27.880916 13878 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45807: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45807: connect: Connection refused (error 111)
Sep 25 16:35:27 dist-test-slave-65pf krb5kdc[13722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354127, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 16:35:27.889328 13883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:27.879917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57844 (local address 127.0.98.61:44969)
0925 16:35:27.880060 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 16:35:27.880067 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 16:35:27.880084 (+ 17us) server_negotiation.cc:407] Connection header received
0925 16:35:27.880124 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:35:27.880129 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:35:27.880183 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:35:27.881341 (+ 1158us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:35:27.881448 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:27.882298 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:27.883151 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:27.884213 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:27.884343 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:27.886408 (+ 2065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:35:27.886411 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:35:27.886413 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:35:27.886433 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:35:27.887926 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:27.888358 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:27.888360 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:27.888361 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:27.888427 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:27.888677 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:27.888679 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:27.888680 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:27.888829 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:35:27.888901 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 16:35:27.889148 (+ 247us) server_negotiation.cc:299] Negotiation successful
0925 16:35:27.889231 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 16:35:27.889394 13806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:27.879552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44969 (local address 127.0.0.1:57844)
0925 16:35:27.879685 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:27.879700 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:35:27.879787 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:35:27.880331 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:35:27.880337 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:35:27.880537 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:35:27.881208 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:27.881220 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:27.882414 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:27.882417 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:27.883026 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:27.883035 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:27.884323 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:27.884326 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:27.884401 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:27.884884 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:35:27.884906 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:35:27.886282 (+ 1376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:35:27.888024 (+ 1742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:35:27.888027 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:35:27.888029 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:35:27.888267 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:35:27.888506 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:35:27.888509 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:35:27.888511 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:35:27.888596 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:35:27.888909 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:35:27.888912 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:35:27.889012 (+ 100us) client_negotiation.cc:769] Sending connection context
0925 16:35:27.889113 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 16:35:27.889240 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 16:35:27.890054 13803 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45807 } has no permanent_uuid. Determining permanent_uuid...
I20260925 16:35:27.890762 13806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:27.890488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45807 (local address 127.0.0.1:37228)
0925 16:35:27.890645 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:27.890699 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 16:35:27.891484 13806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:27.891230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45807 (local address 127.0.0.1:37236)
0925 16:35:27.891380 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:27.891429 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260925 16:35:27.891637 13803 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45807: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45807: connect: Connection refused (error 111)
I20260925 16:35:27.922847 13803 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45807 } attempt: 1
I20260925 16:35:27.923746 13806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:27.923397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45807 (local address 127.0.0.1:37240)
0925 16:35:27.923570 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:27.923648 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260925 16:35:27.924520 13806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:27.924298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45807 (local address 127.0.0.1:37254)
0925 16:35:27.924406 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:27.924460 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260925 16:35:27.924692 13803 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45807: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45807: connect: Connection refused (error 111)
I20260925 16:35:27.930192 13878 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45807 } attempt: 1
I20260925 16:35:27.930964 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:27.930714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45807 (local address 127.0.0.1:37264)
0925 16:35:27.930846 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:27.930899 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 16:35:27.931622 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:27.931429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45807 (local address 127.0.0.1:37276)
0925 16:35:27.931532 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:27.931579 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 16:35:27.931742 13878 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45807: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45807: connect: Connection refused (error 111)
W20260925 16:35:27.986605 13884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:35:27.986886 13884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:35:27.986965 13884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:35:27.990859 13884 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:35:27.990936 13884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:35:27.990970 13884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:35:27.991020 13884 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:35:27.991057 13884 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:35:27.996114 13884 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353966814071-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353966814071-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40127,127.0.98.61:44969,127.0.98.60:45807
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353966814071-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353966814071-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353966814071-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:35:27.997105 13884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:35:27.998099 13884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 16:35:27.999877 13803 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45807 } attempt: 2
I20260925 16:35:27.999905 13878 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45807 } attempt: 2
I20260925 16:35:28.000686 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:28.000392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45807 (local address 127.0.0.1:37288)
0925 16:35:28.000525 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:28.000595 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 16:35:28.000780 13806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:28.000402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45807 (local address 127.0.0.1:37286)
0925 16:35:28.000614 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:28.000691 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
I20260925 16:35:28.001482 13806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:28.001265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45807 (local address 127.0.0.1:37308)
0925 16:35:28.001372 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:28.001421 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 16:35:28.001511 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:28.001273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45807 (local address 127.0.0.1:37306)
0925 16:35:28.001407 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:28.001451 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260925 16:35:28.001626 13803 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45807: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45807: connect: Connection refused (error 111)
W20260925 16:35:28.001658 13878 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45807: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45807: connect: Connection refused (error 111)
W20260925 16:35:28.006723 13893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:35:28.006723 13891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:35:28.006736 13890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:35:28.007027 13884 server_base.cc:1034] running on GCE node
I20260925 16:35:28.007651 13884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:35:28.008719 13884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:35:28.009907 13884 hybrid_clock.cc:648] HybridClock initialized: now 1790354128009893 us; error 29 us; skew 500 ppm
Sep 25 16:35:28 dist-test-slave-65pf krb5kdc[13722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354128, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:35:28.012673 13884 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260925 16:35:28.013572 13884 webserver.cc:460] Webserver started at http://127.0.98.60:39981/ using document root <none> and password file <none>
I20260925 16:35:28.014148 13884 fs_manager.cc:359] Metadata directory not provided
I20260925 16:35:28.014241 13884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:35:28.014479 13884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:35:28.016577 13884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353966814071-392-0/minicluster-data/master-2/data/instance:
uuid: "cdb98f2eef82490598a8a44f0730febc"
format_stamp: "Formatted at 2026-09-25 16:35:28 on dist-test-slave-65pf"
server_key: "fc4f239007630238b29db91f274402bb"
server_key_iv: "775b6902fb4d37ee34028be8823227d8"
server_key_version: "encryptionkey@0"
I20260925 16:35:28.017166 13884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353966814071-392-0/minicluster-data/master-2/wal/instance:
uuid: "cdb98f2eef82490598a8a44f0730febc"
format_stamp: "Formatted at 2026-09-25 16:35:28 on dist-test-slave-65pf"
server_key: "fc4f239007630238b29db91f274402bb"
server_key_iv: "775b6902fb4d37ee34028be8823227d8"
server_key_version: "encryptionkey@0"
I20260925 16:35:28.020989 13884 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 16:35:28.023288 13900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:35:28.024507 13884 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:35:28.024631 13884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353966814071-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353966814071-392-0/minicluster-data/master-2/wal
uuid: "cdb98f2eef82490598a8a44f0730febc"
format_stamp: "Formatted at 2026-09-25 16:35:28 on dist-test-slave-65pf"
server_key: "fc4f239007630238b29db91f274402bb"
server_key_iv: "775b6902fb4d37ee34028be8823227d8"
server_key_version: "encryptionkey@0"
I20260925 16:35:28.024734 13884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353966814071-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353966814071-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353966814071-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:35:28.046859 13884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:35:28.050367 13884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:35:28.050581 13884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:35:28.057686 13884 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:45807
I20260925 16:35:28.057698 13952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:45807 every 8 connection(s)
I20260925 16:35:28.058655 13884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353966814071-392-0/minicluster-data/master-2/data/info.pb
I20260925 16:35:28.060428 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 13884
I20260925 16:35:28.060539 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353966814071-392-0/minicluster-data/master-2/wal/instance
I20260925 16:35:28.060897 392 external_mini_cluster.cc:1428] Setting key d66509ba2d49281298b793350d6e2891
I20260925 16:35:28.061630 13953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:35:28.066298 13953 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40127 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 16:35:28 dist-test-slave-65pf krb5kdc[13722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 25 16:35:28 dist-test-slave-65pf krb5kdc[13722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354128, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:35:28.076071 13956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:28.062274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37324 (local address 127.0.98.60:45807)
0925 16:35:28.062704 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0925 16:35:28.062722 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 16:35:28.062762 (+ 40us) server_negotiation.cc:407] Connection header received
0925 16:35:28.063607 (+ 845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:35:28.063632 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:35:28.064048 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:35:28.064410 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:35:28.065182 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:28.066721 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:28.067639 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:28.068179 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:28.068333 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:28.071283 (+ 2950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:35:28.071288 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:35:28.071296 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:35:28.071320 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:35:28.073088 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:28.073502 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:28.073508 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:28.073513 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:28.073661 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:28.073948 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:28.073953 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:28.073955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:28.074481 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:35:28.074711 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0925 16:35:28.074971 (+ 260us) server_negotiation.cc:299] Negotiation successful
0925 16:35:28.075236 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":112,"threads_started":1}
I20260925 16:35:28.077410 13957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:28.068114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40127 (local address 127.0.0.1:59562)
0925 16:35:28.068567 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:28.068613 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 16:35:28.068728 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:35:28.069040 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:35:28.069046 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:35:28.069529 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:35:28.069725 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:28.069737 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:28.070708 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:28.070714 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:28.071401 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:28.071410 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:28.071864 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:28.071867 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:28.071983 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:28.072560 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:35:28.072571 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:35:28.073958 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:35:28.075772 (+ 1814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:35:28.075779 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:35:28.075784 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:35:28.076050 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:35:28.076381 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:35:28.076384 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:35:28.076386 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:35:28.076473 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:35:28.076897 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:35:28.076901 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:35:28.076998 (+ 97us) client_negotiation.cc:769] Sending connection context
0925 16:35:28.077104 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 16:35:28.077256 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":112,"threads_started":1}
I20260925 16:35:28.077440 13807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:28.068178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59562 (local address 127.0.98.62:40127)
0925 16:35:28.068340 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 16:35:28.068345 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:35:28.068704 (+ 359us) server_negotiation.cc:407] Connection header received
0925 16:35:28.068824 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:35:28.068828 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:35:28.068890 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:35:28.068974 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:35:28.069854 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:28.070536 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:28.071523 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:28.071739 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:28.071862 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:28.074086 (+ 2224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:35:28.074090 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:35:28.074092 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:35:28.074116 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:35:28.075667 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:28.076195 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:28.076201 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:28.076204 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:28.076308 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:28.076567 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:28.076573 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:28.076575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:28.076799 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:35:28.076897 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 16:35:28.077201 (+ 304us) server_negotiation.cc:299] Negotiation successful
0925 16:35:28.077327 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 16:35:28.078999 13953 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44969 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 16:35:28 dist-test-slave-65pf krb5kdc[13722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 25 16:35:28 dist-test-slave-65pf krb5kdc[13722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354128, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 16:35:28.087105 13807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:28.078674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59572 (local address 127.0.98.62:40127)
0925 16:35:28.078806 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 16:35:28.078810 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:35:28.078821 (+ 11us) server_negotiation.cc:407] Connection header received
0925 16:35:28.078851 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:35:28.078853 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:35:28.078888 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:35:28.078958 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:35:28.079710 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:28.080331 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:28.081148 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:28.081340 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:28.081413 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:28.084123 (+ 2710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:35:28.084127 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:35:28.084130 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:35:28.084160 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:35:28.085494 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:28.085934 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:28.085939 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:28.085941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:28.086017 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:28.086281 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:28.086286 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:28.086288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:28.086449 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:35:28.086708 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0925 16:35:28.086833 (+ 125us) server_negotiation.cc:299] Negotiation successful
0925 16:35:28.086947 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 16:35:28.087399 13957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:28.079476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44969 (local address 127.0.0.1:57850)
0925 16:35:28.079635 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:28.079650 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:35:28.079714 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:35:28.079946 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:35:28.079948 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:35:28.080066 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:35:28.080144 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:28.080148 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:28.080887 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:28.080890 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:28.081553 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:28.081562 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:28.082136 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:28.082140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:28.082247 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:28.082637 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:35:28.082654 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:35:28.084169 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:35:28.085964 (+ 1795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:35:28.085968 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:35:28.085970 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:35:28.086159 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:35:28.086426 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:35:28.086431 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:35:28.086432 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:35:28.086522 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:35:28.086906 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:35:28.086909 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:35:28.086998 (+ 89us) client_negotiation.cc:769] Sending connection context
0925 16:35:28.087133 (+ 135us) client_negotiation.cc:240] Negotiation successful
0925 16:35:28.087252 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260925 16:35:28.087430 13883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:28.079589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57850 (local address 127.0.98.61:44969)
0925 16:35:28.079707 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 16:35:28.079711 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:35:28.079725 (+ 14us) server_negotiation.cc:407] Connection header received
0925 16:35:28.079797 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:35:28.079800 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:35:28.079845 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:35:28.079928 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:35:28.080248 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:28.080786 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:28.081664 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:28.082026 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:28.082104 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:28.084307 (+ 2203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:35:28.084310 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:35:28.084312 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:35:28.084352 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:35:28.085837 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:28.086255 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:28.086259 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:28.086261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:28.086336 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:28.086637 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:28.086640 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:28.086642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:28.086806 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:35:28.086914 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 16:35:28.087188 (+ 274us) server_negotiation.cc:299] Negotiation successful
0925 16:35:28.087291 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 16:35:28.088577 13953 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45807 } has no permanent_uuid. Determining permanent_uuid...
I20260925 16:35:28.092813 13878 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45807 } attempt: 3
Sep 25 16:35:28 dist-test-slave-65pf krb5kdc[13722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354128, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 25 16:35:28 dist-test-slave-65pf krb5kdc[13722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 16:35:28.097292 13957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:28.089066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45807 (local address 127.0.0.1:37332)
0925 16:35:28.089280 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:28.089289 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 16:35:28.089380 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:35:28.089583 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:35:28.089585 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:35:28.089736 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:35:28.089791 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:28.089796 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:28.090481 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:28.090484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:28.091036 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:28.091044 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:28.091486 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:28.091489 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:28.091819 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:28.092229 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:35:28.092249 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:35:28.093768 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:35:28.095798 (+ 2030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:35:28.095801 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:35:28.095803 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:35:28.095984 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:35:28.096269 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:35:28.096272 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:35:28.096274 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:35:28.096351 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:35:28.096761 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:35:28.096764 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:35:28.096845 (+ 81us) client_negotiation.cc:769] Sending connection context
0925 16:35:28.096947 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 16:35:28.097080 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":5}
I20260925 16:35:28.097292 13956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:28.089138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37332 (local address 127.0.98.60:45807)
0925 16:35:28.089239 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0925 16:35:28.089242 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:35:28.089347 (+ 105us) server_negotiation.cc:407] Connection header received
0925 16:35:28.089456 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:35:28.089459 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:35:28.089490 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:35:28.089574 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:35:28.089893 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:28.090371 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:28.091143 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:28.091375 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:28.091478 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:28.093900 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:35:28.093903 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:35:28.093904 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:35:28.093921 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:35:28.095702 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:28.096098 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:28.096101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:28.096102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:28.096176 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:28.096435 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:28.096441 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:28.096443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:28.096623 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:35:28.096730 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 16:35:28.097004 (+ 274us) server_negotiation.cc:299] Negotiation successful
0925 16:35:28.097129 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":33}
I20260925 16:35:28.098644 13883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:28.088537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57852 (local address 127.0.98.61:44969)
0925 16:35:28.088654 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 16:35:28.088659 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:35:28.088709 (+ 50us) server_negotiation.cc:407] Connection header received
0925 16:35:28.088842 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:35:28.088845 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:35:28.088886 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:35:28.088981 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:35:28.089635 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:28.090116 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:28.090948 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:28.091179 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:28.091305 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:28.094298 (+ 2993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:35:28.094302 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:35:28.094305 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:35:28.094328 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:35:28.096212 (+ 1884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:28.096738 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:28.096744 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:28.096747 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:28.096827 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:28.097156 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:28.097160 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:28.097161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:28.097413 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:35:28.097643 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0925 16:35:28.098305 (+ 662us) server_negotiation.cc:299] Negotiation successful
0925 16:35:28.098422 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":48}
Sep 25 16:35:28 dist-test-slave-65pf krb5kdc[13722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354127, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 16:35:28.100776 13803 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45807 } attempt: 3
I20260925 16:35:28.101070 13953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdb98f2eef82490598a8a44f0730febc: Bootstrap starting.
I20260925 16:35:28.102185 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:35:28.103602 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:28.093179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45807 (local address 127.0.0.1:37338)
0925 16:35:28.093353 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:28.093365 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 16:35:28.093429 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:35:28.094394 (+ 965us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:35:28.094398 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:35:28.094567 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:35:28.094685 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:28.094691 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:28.095798 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:28.095802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:28.096603 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:28.096612 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:28.097278 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:28.097282 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:28.097763 (+ 481us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:28.098193 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:35:28.098210 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:35:28.099541 (+ 1331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:35:28.101881 (+ 2340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:35:28.101885 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:35:28.101887 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:35:28.102151 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:35:28.102528 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:35:28.102534 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:35:28.102539 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:35:28.102674 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:35:28.103101 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:35:28.103104 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:35:28.103192 (+ 88us) client_negotiation.cc:769] Sending connection context
0925 16:35:28.103283 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 16:35:28.103404 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":377}
I20260925 16:35:28.103935 13953 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cdb98f2eef82490598a8a44f0730febc: Neither blocks nor log segments found. Creating new log.
I20260925 16:35:28.104205 13958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:28.093253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37338 (local address 127.0.98.60:45807)
0925 16:35:28.093860 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0925 16:35:28.093870 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 16:35:28.093886 (+ 16us) server_negotiation.cc:407] Connection header received
0925 16:35:28.093926 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:35:28.093931 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:35:28.093985 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:35:28.094091 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:35:28.094800 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:28.095655 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:28.096756 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:28.097149 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:28.097332 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:28.099681 (+ 2349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:35:28.099685 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:35:28.099687 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:35:28.099711 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:35:28.101753 (+ 2042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:28.102290 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:28.102297 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:28.102300 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:28.102407 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:28.102799 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:28.102802 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:28.102803 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:28.102984 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:35:28.103837 (+ 853us) server_negotiation.cc:1035] Waiting for connection context
0925 16:35:28.103916 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 16:35:28.104027 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":118,"threads_started":1}
I20260925 16:35:28.104854 13953 log.cc:824] T 00000000000000000000000000000000 P cdb98f2eef82490598a8a44f0730febc: Log is configured to *not* fsync() on all Append() calls
Sep 25 16:35:28 dist-test-slave-65pf krb5kdc[13722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354127, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 16:35:28.107137 13953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdb98f2eef82490598a8a44f0730febc: No bootstrap required, opened a new log
I20260925 16:35:28.107307 13878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01d9bb013eee4cdbae694f9a28aa4da3: Bootstrap starting.
I20260925 16:35:28.110219 13878 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01d9bb013eee4cdbae694f9a28aa4da3: Neither blocks nor log segments found. Creating new log.
I20260925 16:35:28.111075 13806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:28.101194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45807 (local address 127.0.0.1:37352)
0925 16:35:28.101547 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:28.101563 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:35:28.101648 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:35:28.102168 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:35:28.102174 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:35:28.102363 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:35:28.102495 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:28.102507 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:28.103555 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:28.103558 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:28.104331 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:28.104347 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:28.104959 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:28.104963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:28.105053 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:28.105547 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:35:28.105564 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:35:28.107019 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:35:28.109218 (+ 2199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:35:28.109222 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:35:28.109224 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:35:28.109499 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:35:28.109857 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:35:28.109861 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:35:28.109863 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:35:28.109987 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:35:28.110494 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:35:28.110499 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:35:28.110631 (+ 132us) client_negotiation.cc:769] Sending connection context
0925 16:35:28.110726 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 16:35:28.110874 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255}
I20260925 16:35:28.111114 13878 log.cc:824] T 00000000000000000000000000000000 P 01d9bb013eee4cdbae694f9a28aa4da3: Log is configured to *not* fsync() on all Append() calls
I20260925 16:35:28.111443 13953 raft_consensus.cc:348] T 00000000000000000000000000000000 P cdb98f2eef82490598a8a44f0730febc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a62985747e49a7beeb56ba4727006b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40127 } } peers { permanent_uuid: "01d9bb013eee4cdbae694f9a28aa4da3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44969 } } peers { permanent_uuid: "cdb98f2eef82490598a8a44f0730febc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45807 } }
I20260925 16:35:28.111694 13953 raft_consensus.cc:374] T 00000000000000000000000000000000 P cdb98f2eef82490598a8a44f0730febc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:35:28.111784 13953 raft_consensus.cc:729] T 00000000000000000000000000000000 P cdb98f2eef82490598a8a44f0730febc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdb98f2eef82490598a8a44f0730febc, State: Initialized, Role: FOLLOWER
I20260925 16:35:28.112353 13956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:28.101807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37352 (local address 127.0.98.60:45807)
0925 16:35:28.101916 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0925 16:35:28.101923 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 16:35:28.101939 (+ 16us) server_negotiation.cc:407] Connection header received
0925 16:35:28.101978 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:35:28.101981 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:35:28.102022 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:35:28.102663 (+ 641us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:35:28.102785 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:28.103427 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:28.104477 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:28.104816 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:28.106156 (+ 1340us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:28.107184 (+ 1028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:35:28.107188 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:35:28.107191 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:35:28.107216 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:35:28.109052 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:28.109638 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:28.109642 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:28.109644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:28.109727 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:28.110123 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:28.110127 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:28.110131 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:28.110351 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:35:28.111980 (+ 1629us) server_negotiation.cc:1035] Waiting for connection context
0925 16:35:28.112068 (+ 88us) server_negotiation.cc:299] Negotiation successful
0925 16:35:28.112190 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 16:35:28.112352 13953 consensus_queue.cc:260] T 00000000000000000000000000000000 P cdb98f2eef82490598a8a44f0730febc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a62985747e49a7beeb56ba4727006b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40127 } } peers { permanent_uuid: "01d9bb013eee4cdbae694f9a28aa4da3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44969 } } peers { permanent_uuid: "cdb98f2eef82490598a8a44f0730febc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45807 } }
I20260925 16:35:28.113361 13969 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdb98f2eef82490598a8a44f0730febc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a62985747e49a7beeb56ba4727006b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40127 } } peers { permanent_uuid: "01d9bb013eee4cdbae694f9a28aa4da3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44969 } } peers { permanent_uuid: "cdb98f2eef82490598a8a44f0730febc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45807 } } }
I20260925 16:35:28.113620 13969 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdb98f2eef82490598a8a44f0730febc [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:35:28.113634 13878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01d9bb013eee4cdbae694f9a28aa4da3: No bootstrap required, opened a new log
I20260925 16:35:28.114655 13953 sys_catalog.cc:564] T 00000000000000000000000000000000 P cdb98f2eef82490598a8a44f0730febc [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:35:28.116384 13807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:28.104913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59580 (local address 127.0.98.62:40127)
0925 16:35:28.105020 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0925 16:35:28.105025 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:35:28.107318 (+ 2293us) server_negotiation.cc:407] Connection header received
0925 16:35:28.107502 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:35:28.107505 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:35:28.107545 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:35:28.107592 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:35:28.108546 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:28.109301 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:28.110436 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:28.110805 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:28.110882 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:28.112746 (+ 1864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:35:28.112752 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:35:28.112757 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:35:28.112787 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:35:28.114548 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:28.115087 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:28.115091 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:28.115092 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:28.115184 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:28.115475 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:28.115478 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:28.115480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:28.115676 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:35:28.115772 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 16:35:28.116110 (+ 338us) server_negotiation.cc:299] Negotiation successful
0925 16:35:28.116234 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 16:35:28.120576 13803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34a62985747e49a7beeb56ba4727006b: Bootstrap starting.
I20260925 16:35:28.118853 13884 master_runner.cc:186] Error getting master registration for 127.0.98.62:40127: OK, instance_id { permanent_uuid: "34a62985747e49a7beeb56ba4727006b" instance_seqno: 1790354127637013 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 16:35:28.122049 13958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:28.105636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37360 (local address 127.0.98.60:45807)
0925 16:35:28.107609 (+ 1973us) server_negotiation.cc:206] Beginning negotiation
0925 16:35:28.107614 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:35:28.112918 (+ 5304us) server_negotiation.cc:407] Connection header received
0925 16:35:28.112959 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:35:28.112976 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:35:28.113028 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:35:28.113104 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:35:28.114283 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:28.115096 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:28.116178 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:28.116540 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:28.116630 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:28.118366 (+ 1736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:35:28.118372 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:35:28.118378 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:35:28.118410 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:35:28.120220 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:28.120737 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:28.120740 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:28.120742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:28.120821 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:28.121091 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:28.121094 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:28.121096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:28.121299 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:35:28.121410 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 16:35:28.121755 (+ 345us) server_negotiation.cc:299] Negotiation successful
0925 16:35:28.121878 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1054,"thread_start_us":100,"threads_started":1}
I20260925 16:35:28.122287 13884 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 16:35:28.123456 13878 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01d9bb013eee4cdbae694f9a28aa4da3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a62985747e49a7beeb56ba4727006b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40127 } } peers { permanent_uuid: "01d9bb013eee4cdbae694f9a28aa4da3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44969 } } peers { permanent_uuid: "cdb98f2eef82490598a8a44f0730febc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45807 } }
I20260925 16:35:28.123799 13878 raft_consensus.cc:374] T 00000000000000000000000000000000 P 01d9bb013eee4cdbae694f9a28aa4da3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:35:28.123901 13803 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34a62985747e49a7beeb56ba4727006b: Neither blocks nor log segments found. Creating new log.
I20260925 16:35:28.123908 13878 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01d9bb013eee4cdbae694f9a28aa4da3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01d9bb013eee4cdbae694f9a28aa4da3, State: Initialized, Role: FOLLOWER
I20260925 16:35:28.124470 13878 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01d9bb013eee4cdbae694f9a28aa4da3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a62985747e49a7beeb56ba4727006b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40127 } } peers { permanent_uuid: "01d9bb013eee4cdbae694f9a28aa4da3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44969 } } peers { permanent_uuid: "cdb98f2eef82490598a8a44f0730febc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45807 } }
I20260925 16:35:28.124886 13803 log.cc:824] T 00000000000000000000000000000000 P 34a62985747e49a7beeb56ba4727006b: Log is configured to *not* fsync() on all Append() calls
I20260925 16:35:28.125015 13981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01d9bb013eee4cdbae694f9a28aa4da3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a62985747e49a7beeb56ba4727006b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40127 } } peers { permanent_uuid: "01d9bb013eee4cdbae694f9a28aa4da3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44969 } } peers { permanent_uuid: "cdb98f2eef82490598a8a44f0730febc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45807 } } }
I20260925 16:35:28.125236 13981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01d9bb013eee4cdbae694f9a28aa4da3 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:35:28.125911 13878 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01d9bb013eee4cdbae694f9a28aa4da3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:35:28.126905 13883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:28.111615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57868 (local address 127.0.98.61:44969)
0925 16:35:28.111735 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 16:35:28.111740 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:35:28.118291 (+ 6551us) server_negotiation.cc:407] Connection header received
0925 16:35:28.118461 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:35:28.118464 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:35:28.118510 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:35:28.118567 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:35:28.119447 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:28.120232 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:28.121293 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:28.121603 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:28.121686 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:28.123370 (+ 1684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:35:28.123374 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:35:28.123376 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:35:28.123402 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:35:28.125037 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:28.125540 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:28.125544 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:28.125546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:28.125628 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:28.125991 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:28.125994 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:28.125996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:28.126201 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:35:28.126327 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 16:35:28.126665 (+ 338us) server_negotiation.cc:299] Negotiation successful
0925 16:35:28.126775 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260925 16:35:28.128379 13980 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cdb98f2eef82490598a8a44f0730febc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 16:35:28.128533 13980 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 16:35:28.131232 13803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34a62985747e49a7beeb56ba4727006b: No bootstrap required, opened a new log
W20260925 16:35:28.131950 13994 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 01d9bb013eee4cdbae694f9a28aa4da3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 16:35:28.132033 13994 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 16:35:28.130172 13810 master_runner.cc:186] Error getting master registration for 127.0.98.62:40127: OK, instance_id { permanent_uuid: "34a62985747e49a7beeb56ba4727006b" instance_seqno: 1790354127637013 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 16:35:28.135547 13810 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 16:35:28.135427 13803 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34a62985747e49a7beeb56ba4727006b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a62985747e49a7beeb56ba4727006b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40127 } } peers { permanent_uuid: "01d9bb013eee4cdbae694f9a28aa4da3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44969 } } peers { permanent_uuid: "cdb98f2eef82490598a8a44f0730febc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45807 } }
I20260925 16:35:28.135651 13803 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34a62985747e49a7beeb56ba4727006b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:35:28.135702 13803 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34a62985747e49a7beeb56ba4727006b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34a62985747e49a7beeb56ba4727006b, State: Initialized, Role: FOLLOWER
I20260925 16:35:28.136373 13803 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34a62985747e49a7beeb56ba4727006b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a62985747e49a7beeb56ba4727006b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40127 } } peers { permanent_uuid: "01d9bb013eee4cdbae694f9a28aa4da3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44969 } } peers { permanent_uuid: "cdb98f2eef82490598a8a44f0730febc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45807 } }
I20260925 16:35:28.137810 13803 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34a62985747e49a7beeb56ba4727006b [sys.catalog]: configured and running, proceeding with master startup.
W20260925 16:35:28.144557 14007 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 34a62985747e49a7beeb56ba4727006b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 16:35:28.144797 14007 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 16:35:28.147734 13735 master_runner.cc:263] No leader master found from master 34a62985747e49a7beeb56ba4727006b
I20260925 16:35:28.147820 13735 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 16:35:28.150816 13996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34a62985747e49a7beeb56ba4727006b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a62985747e49a7beeb56ba4727006b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40127 } } peers { permanent_uuid: "01d9bb013eee4cdbae694f9a28aa4da3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44969 } } peers { permanent_uuid: "cdb98f2eef82490598a8a44f0730febc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45807 } } }
I20260925 16:35:28.151003 13996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34a62985747e49a7beeb56ba4727006b [sys.catalog]: This master's current role is: FOLLOWER
W20260925 16:35:28.302064 13906 tablet.cc:2369] T 00000000000000000000000000000000 P cdb98f2eef82490598a8a44f0730febc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 16:35:28.338766 13831 tablet.cc:2369] T 00000000000000000000000000000000 P 01d9bb013eee4cdbae694f9a28aa4da3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:35:28.379753 13981 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01d9bb013eee4cdbae694f9a28aa4da3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:35:28.379936 13981 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01d9bb013eee4cdbae694f9a28aa4da3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a62985747e49a7beeb56ba4727006b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40127 } } peers { permanent_uuid: "01d9bb013eee4cdbae694f9a28aa4da3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44969 } } peers { permanent_uuid: "cdb98f2eef82490598a8a44f0730febc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45807 } }
I20260925 16:35:28.380821 13981 leader_election.cc:288] T 00000000000000000000000000000000 P 01d9bb013eee4cdbae694f9a28aa4da3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 34a62985747e49a7beeb56ba4727006b (127.0.98.62:40127), cdb98f2eef82490598a8a44f0730febc (127.0.98.60:45807)
I20260925 16:35:28.381371 13927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "01d9bb013eee4cdbae694f9a28aa4da3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdb98f2eef82490598a8a44f0730febc" is_pre_election: true
I20260925 16:35:28.381372 13777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "01d9bb013eee4cdbae694f9a28aa4da3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34a62985747e49a7beeb56ba4727006b" is_pre_election: true
I20260925 16:35:28.381687 13927 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cdb98f2eef82490598a8a44f0730febc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01d9bb013eee4cdbae694f9a28aa4da3 in term 0.
I20260925 16:35:28.381701 13777 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 34a62985747e49a7beeb56ba4727006b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01d9bb013eee4cdbae694f9a28aa4da3 in term 0.
I20260925 16:35:28.382140 13827 leader_election.cc:302] T 00000000000000000000000000000000 P 01d9bb013eee4cdbae694f9a28aa4da3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01d9bb013eee4cdbae694f9a28aa4da3, cdb98f2eef82490598a8a44f0730febc; no voters:
I20260925 16:35:28.382427 13981 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01d9bb013eee4cdbae694f9a28aa4da3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:35:28.382512 13981 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01d9bb013eee4cdbae694f9a28aa4da3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:35:28.382556 13981 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01d9bb013eee4cdbae694f9a28aa4da3 [term 0 FOLLOWER]: Advancing to term 1
W20260925 16:35:28.383674 13756 tablet.cc:2369] T 00000000000000000000000000000000 P 34a62985747e49a7beeb56ba4727006b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:35:28.383731 13981 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01d9bb013eee4cdbae694f9a28aa4da3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a62985747e49a7beeb56ba4727006b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40127 } } peers { permanent_uuid: "01d9bb013eee4cdbae694f9a28aa4da3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44969 } } peers { permanent_uuid: "cdb98f2eef82490598a8a44f0730febc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45807 } }
I20260925 16:35:28.384063 13981 leader_election.cc:288] T 00000000000000000000000000000000 P 01d9bb013eee4cdbae694f9a28aa4da3 [CANDIDATE]: Term 1 election: Requested vote from peers 34a62985747e49a7beeb56ba4727006b (127.0.98.62:40127), cdb98f2eef82490598a8a44f0730febc (127.0.98.60:45807)
I20260925 16:35:28.384374 13777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "01d9bb013eee4cdbae694f9a28aa4da3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34a62985747e49a7beeb56ba4727006b"
I20260925 16:35:28.384413 13927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "01d9bb013eee4cdbae694f9a28aa4da3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdb98f2eef82490598a8a44f0730febc"
I20260925 16:35:28.384500 13777 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34a62985747e49a7beeb56ba4727006b [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:35:28.384518 13927 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cdb98f2eef82490598a8a44f0730febc [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:35:28.385711 13927 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cdb98f2eef82490598a8a44f0730febc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01d9bb013eee4cdbae694f9a28aa4da3 in term 1.
I20260925 16:35:28.385711 13777 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 34a62985747e49a7beeb56ba4727006b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01d9bb013eee4cdbae694f9a28aa4da3 in term 1.
I20260925 16:35:28.386001 13827 leader_election.cc:302] T 00000000000000000000000000000000 P 01d9bb013eee4cdbae694f9a28aa4da3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01d9bb013eee4cdbae694f9a28aa4da3, cdb98f2eef82490598a8a44f0730febc; no voters:
I20260925 16:35:28.386271 13981 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01d9bb013eee4cdbae694f9a28aa4da3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:35:28.386485 13981 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01d9bb013eee4cdbae694f9a28aa4da3 [term 1 LEADER]: Becoming Leader. State: Replica: 01d9bb013eee4cdbae694f9a28aa4da3, State: Running, Role: LEADER
I20260925 16:35:28.386813 13981 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01d9bb013eee4cdbae694f9a28aa4da3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a62985747e49a7beeb56ba4727006b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40127 } } peers { permanent_uuid: "01d9bb013eee4cdbae694f9a28aa4da3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44969 } } peers { permanent_uuid: "cdb98f2eef82490598a8a44f0730febc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45807 } }
I20260925 16:35:28.388849 14009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01d9bb013eee4cdbae694f9a28aa4da3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01d9bb013eee4cdbae694f9a28aa4da3. Latest consensus state: current_term: 1 leader_uuid: "01d9bb013eee4cdbae694f9a28aa4da3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a62985747e49a7beeb56ba4727006b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40127 } } peers { permanent_uuid: "01d9bb013eee4cdbae694f9a28aa4da3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44969 } } peers { permanent_uuid: "cdb98f2eef82490598a8a44f0730febc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45807 } } }
I20260925 16:35:28.388964 14009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01d9bb013eee4cdbae694f9a28aa4da3 [sys.catalog]: This master's current role is: LEADER
I20260925 16:35:28.389460 14011 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:35:28.390254 14011 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:35:28.395061 13927 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cdb98f2eef82490598a8a44f0730febc [term 1 FOLLOWER]: Refusing update from remote peer 01d9bb013eee4cdbae694f9a28aa4da3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:35:28.395223 13777 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 34a62985747e49a7beeb56ba4727006b [term 1 FOLLOWER]: Refusing update from remote peer 01d9bb013eee4cdbae694f9a28aa4da3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:35:28.395756 13981 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 01d9bb013eee4cdbae694f9a28aa4da3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdb98f2eef82490598a8a44f0730febc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:35:28.395926 14009 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 01d9bb013eee4cdbae694f9a28aa4da3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34a62985747e49a7beeb56ba4727006b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:35:28.399621 13996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34a62985747e49a7beeb56ba4727006b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01d9bb013eee4cdbae694f9a28aa4da3. Latest consensus state: current_term: 1 leader_uuid: "01d9bb013eee4cdbae694f9a28aa4da3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a62985747e49a7beeb56ba4727006b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40127 } } peers { permanent_uuid: "01d9bb013eee4cdbae694f9a28aa4da3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44969 } } peers { permanent_uuid: "cdb98f2eef82490598a8a44f0730febc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45807 } } }
I20260925 16:35:28.399739 13996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34a62985747e49a7beeb56ba4727006b [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:35:28.399703 13969 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdb98f2eef82490598a8a44f0730febc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01d9bb013eee4cdbae694f9a28aa4da3. Latest consensus state: current_term: 1 leader_uuid: "01d9bb013eee4cdbae694f9a28aa4da3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a62985747e49a7beeb56ba4727006b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40127 } } peers { permanent_uuid: "01d9bb013eee4cdbae694f9a28aa4da3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44969 } } peers { permanent_uuid: "cdb98f2eef82490598a8a44f0730febc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45807 } } }
I20260925 16:35:28.399816 13969 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdb98f2eef82490598a8a44f0730febc [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:35:28.402308 14017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01d9bb013eee4cdbae694f9a28aa4da3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "01d9bb013eee4cdbae694f9a28aa4da3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a62985747e49a7beeb56ba4727006b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40127 } } peers { permanent_uuid: "01d9bb013eee4cdbae694f9a28aa4da3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44969 } } peers { permanent_uuid: "cdb98f2eef82490598a8a44f0730febc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45807 } } }
I20260925 16:35:28.402520 14017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01d9bb013eee4cdbae694f9a28aa4da3 [sys.catalog]: This master's current role is: LEADER
I20260925 16:35:28.402621 13996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34a62985747e49a7beeb56ba4727006b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "01d9bb013eee4cdbae694f9a28aa4da3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a62985747e49a7beeb56ba4727006b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40127 } } peers { permanent_uuid: "01d9bb013eee4cdbae694f9a28aa4da3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44969 } } peers { permanent_uuid: "cdb98f2eef82490598a8a44f0730febc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45807 } } }
I20260925 16:35:28.402737 13996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34a62985747e49a7beeb56ba4727006b [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:35:28.402741 14009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01d9bb013eee4cdbae694f9a28aa4da3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "01d9bb013eee4cdbae694f9a28aa4da3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a62985747e49a7beeb56ba4727006b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40127 } } peers { permanent_uuid: "01d9bb013eee4cdbae694f9a28aa4da3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44969 } } peers { permanent_uuid: "cdb98f2eef82490598a8a44f0730febc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45807 } } }
I20260925 16:35:28.402833 14009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01d9bb013eee4cdbae694f9a28aa4da3 [sys.catalog]: This master's current role is: LEADER
I20260925 16:35:28.402663 13969 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdb98f2eef82490598a8a44f0730febc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "01d9bb013eee4cdbae694f9a28aa4da3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a62985747e49a7beeb56ba4727006b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40127 } } peers { permanent_uuid: "01d9bb013eee4cdbae694f9a28aa4da3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44969 } } peers { permanent_uuid: "cdb98f2eef82490598a8a44f0730febc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45807 } } }
I20260925 16:35:28.403230 13969 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdb98f2eef82490598a8a44f0730febc [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:35:28.403414 14013 mvcc.cc:205] Tried to move back new op lower bound from 7333290509899767808 to 7333290509874769920. Current Snapshot: MvccSnapshot[applied={T|T < 7333290509899767808}]
I20260925 16:35:28.404814 14011 catalog_manager.cc:1347] Generated new cluster ID: d9084d0b96224e328f61ddb31a7dbecb
I20260925 16:35:28.404874 14011 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:35:28.423527 14011 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:35:28.424870 14011 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:35:28.434109 14011 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 01d9bb013eee4cdbae694f9a28aa4da3: Generated new TSK 0
I20260925 16:35:28.434988 14011 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:35:29.129951 13980 catalog_manager.cc:1259] Loaded cluster ID: d9084d0b96224e328f61ddb31a7dbecb
I20260925 16:35:29.130065 13980 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cdb98f2eef82490598a8a44f0730febc: loading cluster ID for follower catalog manager: success
I20260925 16:35:29.131793 13980 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cdb98f2eef82490598a8a44f0730febc: acquiring CA information for follower catalog manager: success
I20260925 16:35:29.132979 13980 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cdb98f2eef82490598a8a44f0730febc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 16:35:29.146056 14007 catalog_manager.cc:1259] Loaded cluster ID: d9084d0b96224e328f61ddb31a7dbecb
I20260925 16:35:29.146132 14007 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 34a62985747e49a7beeb56ba4727006b: loading cluster ID for follower catalog manager: success
I20260925 16:35:29.147703 14007 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 34a62985747e49a7beeb56ba4727006b: acquiring CA information for follower catalog manager: success
I20260925 16:35:29.148772 14007 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 34a62985747e49a7beeb56ba4727006b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 16:35:29.510862 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 13735
I20260925 16:35:29.517087 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 13810
I20260925 16:35:29.522650 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 13884
2026-09-25T16:35:29Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3216 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790353966814071-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:35:29 dist-test-slave-65pf krb5kdc[14026](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:35:29 dist-test-slave-65pf krb5kdc[14026](info): set up 2 sockets
Sep 25 16:35:29 dist-test-slave-65pf krb5kdc[14026](info): commencing operation
krb5kdc: starting...
W20260925 16:35:30.566792 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790353966814071-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790353966814071-392-0/krb5kdc/test-user.keytab.
Sep 25 16:35:30 dist-test-slave-65pf krb5kdc[14026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:35:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:35:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 16:35:30.689911 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790353966814071-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790353966814071-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790353966814071-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46499
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46499
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:35:30.799206 14039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:35:30.799507 14039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:35:30.799584 14039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:35:30.803704 14039 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:35:30.803791 14039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:35:30.803833 14039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:35:30.803892 14039 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:35:30.803949 14039 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:35:30.809350 14039 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790353966814071-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790353966814071-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46499
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790353966814071-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:35:30.810412 14039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:35:30.811437 14039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:35:30.819924 14044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:35:30.820010 14039 server_base.cc:1034] running on GCE node
W20260925 16:35:30.819964 14045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:35:30.819947 14047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:35:30.820746 14039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:35:30.821818 14039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:35:30.823007 14039 hybrid_clock.cc:648] HybridClock initialized: now 1790354130822993 us; error 26 us; skew 500 ppm
Sep 25 16:35:30 dist-test-slave-65pf krb5kdc[14026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354130, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:35:30.825641 14039 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 16:35:30.826546 14039 webserver.cc:460] Webserver started at http://127.0.98.62:34161/ using document root <none> and password file <none>
I20260925 16:35:30.827131 14039 fs_manager.cc:359] Metadata directory not provided
I20260925 16:35:30.827224 14039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:35:30.827453 14039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:35:30.829481 14039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790353966814071-392-0/minicluster-data/master-0/data/instance:
uuid: "1c3455006e794d56bcb23bb73ffadbbd"
format_stamp: "Formatted at 2026-09-25 16:35:30 on dist-test-slave-65pf"
server_key: "478da30555f3201faf3b054668e33dce"
server_key_iv: "c36eeb7a0044981d15c74e9f3c84c9f7"
server_key_version: "encryptionkey@0"
I20260925 16:35:30.830049 14039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790353966814071-392-0/minicluster-data/master-0/wal/instance:
uuid: "1c3455006e794d56bcb23bb73ffadbbd"
format_stamp: "Formatted at 2026-09-25 16:35:30 on dist-test-slave-65pf"
server_key: "478da30555f3201faf3b054668e33dce"
server_key_iv: "c36eeb7a0044981d15c74e9f3c84c9f7"
server_key_version: "encryptionkey@0"
I20260925 16:35:30.833726 14039 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260925 16:35:30.836349 14054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:35:30.837592 14039 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 16:35:30.837734 14039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790353966814071-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790353966814071-392-0/minicluster-data/master-0/wal
uuid: "1c3455006e794d56bcb23bb73ffadbbd"
format_stamp: "Formatted at 2026-09-25 16:35:30 on dist-test-slave-65pf"
server_key: "478da30555f3201faf3b054668e33dce"
server_key_iv: "c36eeb7a0044981d15c74e9f3c84c9f7"
server_key_version: "encryptionkey@0"
I20260925 16:35:30.837850 14039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790353966814071-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790353966814071-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790353966814071-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:35:30.847613 14039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:35:30.850414 14039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:35:30.850692 14039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:35:30.857600 14039 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46499
I20260925 16:35:30.857620 14106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46499 every 8 connection(s)
I20260925 16:35:30.858547 14039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:35:30.861241 14107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:35:30.866204 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 14039
I20260925 16:35:30.866320 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790353966814071-392-0/minicluster-data/master-0/wal/instance
I20260925 16:35:30.866657 392 external_mini_cluster.cc:1428] Setting key 6da7892f7fd90a3585112f6c42c917e4
I20260925 16:35:30.867133 14107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c3455006e794d56bcb23bb73ffadbbd: Bootstrap starting.
I20260925 16:35:30.869578 14107 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c3455006e794d56bcb23bb73ffadbbd: Neither blocks nor log segments found. Creating new log.
I20260925 16:35:30.870312 14107 log.cc:824] T 00000000000000000000000000000000 P 1c3455006e794d56bcb23bb73ffadbbd: Log is configured to *not* fsync() on all Append() calls
I20260925 16:35:30.872165 14107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c3455006e794d56bcb23bb73ffadbbd: No bootstrap required, opened a new log
Sep 25 16:35:30 dist-test-slave-65pf krb5kdc[14026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:35:30.875713 14107 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c3455006e794d56bcb23bb73ffadbbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c3455006e794d56bcb23bb73ffadbbd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46499 } }
I20260925 16:35:30.876041 14107 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c3455006e794d56bcb23bb73ffadbbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:35:30.876103 14107 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c3455006e794d56bcb23bb73ffadbbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c3455006e794d56bcb23bb73ffadbbd, State: Initialized, Role: FOLLOWER
I20260925 16:35:30.876667 14107 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c3455006e794d56bcb23bb73ffadbbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c3455006e794d56bcb23bb73ffadbbd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46499 } }
I20260925 16:35:30.876837 14107 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c3455006e794d56bcb23bb73ffadbbd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:35:30.876942 14107 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c3455006e794d56bcb23bb73ffadbbd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:35:30.877058 14107 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c3455006e794d56bcb23bb73ffadbbd [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:35:30.877970 14107 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c3455006e794d56bcb23bb73ffadbbd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c3455006e794d56bcb23bb73ffadbbd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46499 } }
I20260925 16:35:30.878327 14107 leader_election.cc:302] T 00000000000000000000000000000000 P 1c3455006e794d56bcb23bb73ffadbbd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c3455006e794d56bcb23bb73ffadbbd; no voters:
I20260925 16:35:30.878711 14107 leader_election.cc:288] T 00000000000000000000000000000000 P 1c3455006e794d56bcb23bb73ffadbbd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:35:30.878795 14112 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c3455006e794d56bcb23bb73ffadbbd [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:35:30.879004 14112 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c3455006e794d56bcb23bb73ffadbbd [term 1 LEADER]: Becoming Leader. State: Replica: 1c3455006e794d56bcb23bb73ffadbbd, State: Running, Role: LEADER
I20260925 16:35:30.879379 14112 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c3455006e794d56bcb23bb73ffadbbd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c3455006e794d56bcb23bb73ffadbbd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46499 } }
I20260925 16:35:30.880090 14107 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c3455006e794d56bcb23bb73ffadbbd [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:35:30.881013 14113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c3455006e794d56bcb23bb73ffadbbd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c3455006e794d56bcb23bb73ffadbbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c3455006e794d56bcb23bb73ffadbbd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46499 } } }
I20260925 16:35:30.881204 14113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c3455006e794d56bcb23bb73ffadbbd [sys.catalog]: This master's current role is: LEADER
I20260925 16:35:30.881227 14110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:30.868013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60634 (local address 127.0.98.62:46499)
0925 16:35:30.868452 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0925 16:35:30.868467 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 16:35:30.868504 (+ 37us) server_negotiation.cc:407] Connection header received
0925 16:35:30.869118 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:35:30.869135 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:35:30.869461 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:35:30.869799 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:35:30.870638 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:30.871594 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:30.872522 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:30.872993 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:30.873167 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:30.875638 (+ 2471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:35:30.875643 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:35:30.875656 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:35:30.875682 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:35:30.877605 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:30.878064 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:30.878070 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:30.878084 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:30.878244 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:30.878544 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:30.878549 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:30.878551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:30.879151 (+ 600us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:35:30.879350 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0925 16:35:30.879702 (+ 352us) server_negotiation.cc:299] Negotiation successful
0925 16:35:30.880173 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":142,"threads_started":1}
I20260925 16:35:30.882030 14125 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c3455006e794d56bcb23bb73ffadbbd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c3455006e794d56bcb23bb73ffadbbd. Latest consensus state: current_term: 1 leader_uuid: "1c3455006e794d56bcb23bb73ffadbbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c3455006e794d56bcb23bb73ffadbbd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46499 } } }
I20260925 16:35:30.882110 14125 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c3455006e794d56bcb23bb73ffadbbd [sys.catalog]: This master's current role is: LEADER
I20260925 16:35:30.884537 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260925 16:35:30.885107 14127 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1c3455006e794d56bcb23bb73ffadbbd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 16:35:30.885196 14127 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 16:35:30.885299 14128 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:35:30.886067 14128 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:35:30.891446 14128 catalog_manager.cc:1347] Generated new cluster ID: d3962eab875b470fbf2d3eff7e32d095
I20260925 16:35:30.891516 14128 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:35:30.908036 14128 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:35:30.909014 14128 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:35:30.913244 14128 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c3455006e794d56bcb23bb73ffadbbd: Generated new TSK 0
I20260925 16:35:30.913892 14128 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:35:30.940833 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 14039
2026-09-25T16:35:30Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1417 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:35:30 dist-test-slave-65pf krb5kdc[14135](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:35:30 dist-test-slave-65pf krb5kdc[14135](info): set up 2 sockets
Sep 25 16:35:30 dist-test-slave-65pf krb5kdc[14135](info): commencing operation
krb5kdc: starting...
W20260925 16:35:31.982929 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/krb5kdc/test-user.keytab.
Sep 25 16:35:32 dist-test-slave-65pf krb5kdc[14135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:35:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:35:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 16:35:32.099699 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35813
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35813
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:35:32.208700 14148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:35:32.209002 14148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:35:32.209043 14148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:35:32.213183 14148 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:35:32.213249 14148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:35:32.213269 14148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:35:32.213286 14148 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:35:32.213344 14148 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:35:32.218890 14148 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:35:32.219970 14148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:35:32.221009 14148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:35:32.229584 14154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:35:32.229590 14156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:35:32.229637 14148 server_base.cc:1034] running on GCE node
W20260925 16:35:32.229597 14153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:35:32.230322 14148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:35:32.231443 14148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:35:32.232635 14148 hybrid_clock.cc:648] HybridClock initialized: now 1790354132232630 us; error 49 us; skew 500 ppm
Sep 25 16:35:32 dist-test-slave-65pf krb5kdc[14135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354132, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:35:32.235272 14148 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 16:35:32.236117 14148 webserver.cc:460] Webserver started at http://127.0.98.62:34251/ using document root <none> and password file <none>
I20260925 16:35:32.236681 14148 fs_manager.cc:359] Metadata directory not provided
I20260925 16:35:32.236771 14148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:35:32.236999 14148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:35:32.239136 14148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/master-0/data/instance:
uuid: "0b6a310d97744021b7dd800019cc5eef"
format_stamp: "Formatted at 2026-09-25 16:35:32 on dist-test-slave-65pf"
server_key: "47e02e14eb355f1041a158b1faf73cc9"
server_key_iv: "1ea01d8ead1e8ff16019783038effe46"
server_key_version: "encryptionkey@0"
I20260925 16:35:32.239703 14148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/master-0/wal/instance:
uuid: "0b6a310d97744021b7dd800019cc5eef"
format_stamp: "Formatted at 2026-09-25 16:35:32 on dist-test-slave-65pf"
server_key: "47e02e14eb355f1041a158b1faf73cc9"
server_key_iv: "1ea01d8ead1e8ff16019783038effe46"
server_key_version: "encryptionkey@0"
I20260925 16:35:32.243301 14148 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 16:35:32.245703 14163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:35:32.246933 14148 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:35:32.247072 14148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/master-0/wal
uuid: "0b6a310d97744021b7dd800019cc5eef"
format_stamp: "Formatted at 2026-09-25 16:35:32 on dist-test-slave-65pf"
server_key: "47e02e14eb355f1041a158b1faf73cc9"
server_key_iv: "1ea01d8ead1e8ff16019783038effe46"
server_key_version: "encryptionkey@0"
I20260925 16:35:32.247179 14148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:35:32.260470 14148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:35:32.263402 14148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:35:32.263610 14148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:35:32.270635 14148 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35813
I20260925 16:35:32.270877 14215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35813 every 8 connection(s)
I20260925 16:35:32.271612 14148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:35:32.274281 14216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:35:32.275557 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 14148
I20260925 16:35:32.275698 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/master-0/wal/instance
I20260925 16:35:32.276005 392 external_mini_cluster.cc:1428] Setting key 6dca043ec11f753a6b8b729bd0dd16e3
I20260925 16:35:32.280673 14216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b6a310d97744021b7dd800019cc5eef: Bootstrap starting.
I20260925 16:35:32.283192 14216 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b6a310d97744021b7dd800019cc5eef: Neither blocks nor log segments found. Creating new log.
I20260925 16:35:32.284013 14216 log.cc:824] T 00000000000000000000000000000000 P 0b6a310d97744021b7dd800019cc5eef: Log is configured to *not* fsync() on all Append() calls
Sep 25 16:35:32 dist-test-slave-65pf krb5kdc[14135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:35:32.286279 14216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b6a310d97744021b7dd800019cc5eef: No bootstrap required, opened a new log
I20260925 16:35:32.290369 14219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:32.277440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58916 (local address 127.0.98.62:35813)
0925 16:35:32.277838 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0925 16:35:32.277855 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 16:35:32.277897 (+ 42us) server_negotiation.cc:407] Connection header received
0925 16:35:32.278723 (+ 826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:35:32.278748 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:35:32.279149 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:35:32.279506 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:35:32.280126 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:32.281546 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:32.282357 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:32.283033 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:32.283212 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:32.285947 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:35:32.285952 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:35:32.285965 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:35:32.285995 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:35:32.287656 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:32.288135 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:32.288139 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:32.288143 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:32.288245 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:32.288566 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:32.288568 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:32.288569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:32.288920 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:35:32.289061 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 16:35:32.289350 (+ 289us) server_negotiation.cc:299] Negotiation successful
0925 16:35:32.289637 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":100,"threads_started":1}
I20260925 16:35:32.290381 14216 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b6a310d97744021b7dd800019cc5eef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b6a310d97744021b7dd800019cc5eef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35813 } }
I20260925 16:35:32.290752 14216 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b6a310d97744021b7dd800019cc5eef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:35:32.290832 14216 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b6a310d97744021b7dd800019cc5eef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6a310d97744021b7dd800019cc5eef, State: Initialized, Role: FOLLOWER
I20260925 16:35:32.291244 14216 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b6a310d97744021b7dd800019cc5eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b6a310d97744021b7dd800019cc5eef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35813 } }
I20260925 16:35:32.291388 14216 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b6a310d97744021b7dd800019cc5eef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:35:32.291484 14216 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b6a310d97744021b7dd800019cc5eef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:35:32.291586 14216 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b6a310d97744021b7dd800019cc5eef [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:35:32.292665 14216 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b6a310d97744021b7dd800019cc5eef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b6a310d97744021b7dd800019cc5eef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35813 } }
I20260925 16:35:32.292999 14216 leader_election.cc:302] T 00000000000000000000000000000000 P 0b6a310d97744021b7dd800019cc5eef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b6a310d97744021b7dd800019cc5eef; no voters:
I20260925 16:35:32.293289 14216 leader_election.cc:288] T 00000000000000000000000000000000 P 0b6a310d97744021b7dd800019cc5eef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:35:32.293402 14221 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b6a310d97744021b7dd800019cc5eef [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:35:32.293650 14221 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b6a310d97744021b7dd800019cc5eef [term 1 LEADER]: Becoming Leader. State: Replica: 0b6a310d97744021b7dd800019cc5eef, State: Running, Role: LEADER
I20260925 16:35:32.294006 14221 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b6a310d97744021b7dd800019cc5eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b6a310d97744021b7dd800019cc5eef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35813 } }
I20260925 16:35:32.294454 14216 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b6a310d97744021b7dd800019cc5eef [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:35:32.295713 14222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b6a310d97744021b7dd800019cc5eef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b6a310d97744021b7dd800019cc5eef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b6a310d97744021b7dd800019cc5eef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35813 } } }
I20260925 16:35:32.295866 14222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b6a310d97744021b7dd800019cc5eef [sys.catalog]: This master's current role is: LEADER
I20260925 16:35:32.295712 14223 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b6a310d97744021b7dd800019cc5eef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b6a310d97744021b7dd800019cc5eef. Latest consensus state: current_term: 1 leader_uuid: "0b6a310d97744021b7dd800019cc5eef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b6a310d97744021b7dd800019cc5eef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35813 } } }
I20260925 16:35:32.296186 14223 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b6a310d97744021b7dd800019cc5eef [sys.catalog]: This master's current role is: LEADER
I20260925 16:35:32.296250 14232 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:35:32.299100 14232 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:35:32.304085 14232 catalog_manager.cc:1347] Generated new cluster ID: 6adea33195954dbeb67088ce5172c8ed
I20260925 16:35:32.304148 14232 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:35:32.324313 14232 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:35:32.325641 14232 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:35:32.336153 14232 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b6a310d97744021b7dd800019cc5eef: Generated new TSK 0
I20260925 16:35:32.337003 14232 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 16:35:32.367383 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35813
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:35:32.477599 14242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:35:32.477885 14242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:35:32.477931 14242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:35:32.481932 14242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:35:32.482000 14242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:35:32.482059 14242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 16:35:32.487291 14242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:35:32.488394 14242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:35:32.489449 14242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:35:32.499145 14250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:35:32.499195 14248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:35:32.499199 14247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:35:32.499424 14242 server_base.cc:1034] running on GCE node
I20260925 16:35:32.499825 14242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:35:32.500444 14242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:35:32.501606 14242 hybrid_clock.cc:648] HybridClock initialized: now 1790354132501588 us; error 30 us; skew 500 ppm
Sep 25 16:35:32 dist-test-slave-65pf krb5kdc[14135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354132, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:35:32.504340 14242 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 16:35:32.505203 14242 webserver.cc:460] Webserver started at http://127.0.98.1:33983/ using document root <none> and password file <none>
I20260925 16:35:32.505782 14242 fs_manager.cc:359] Metadata directory not provided
I20260925 16:35:32.505879 14242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:35:32.506111 14242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:35:32.508242 14242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-0/data/instance:
uuid: "e58d56335e084712b0f4c8aecff74e8c"
format_stamp: "Formatted at 2026-09-25 16:35:32 on dist-test-slave-65pf"
server_key: "e8d8a16ebfeb835b45e1ac4d80f7e1a3"
server_key_iv: "4ac91e6f2b8d6148b67d33911daf5d94"
server_key_version: "encryptionkey@0"
I20260925 16:35:32.508810 14242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e58d56335e084712b0f4c8aecff74e8c"
format_stamp: "Formatted at 2026-09-25 16:35:32 on dist-test-slave-65pf"
server_key: "e8d8a16ebfeb835b45e1ac4d80f7e1a3"
server_key_iv: "4ac91e6f2b8d6148b67d33911daf5d94"
server_key_version: "encryptionkey@0"
I20260925 16:35:32.512529 14242 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 16:35:32.515128 14257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:35:32.516393 14242 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:35:32.516532 14242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-0/wal
uuid: "e58d56335e084712b0f4c8aecff74e8c"
format_stamp: "Formatted at 2026-09-25 16:35:32 on dist-test-slave-65pf"
server_key: "e8d8a16ebfeb835b45e1ac4d80f7e1a3"
server_key_iv: "4ac91e6f2b8d6148b67d33911daf5d94"
server_key_version: "encryptionkey@0"
I20260925 16:35:32.516644 14242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:35:32.529332 14242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:35:32.532177 14242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:35:32.532377 14242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:35:32.532943 14242 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:35:32.533771 14242 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:35:32.533849 14242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:35:32.533912 14242 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:35:32.533957 14242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:35:32.542878 14242 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38835
I20260925 16:35:32.542920 14370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38835 every 8 connection(s)
I20260925 16:35:32.543860 14242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
Sep 25 16:35:32 dist-test-slave-65pf krb5kdc[14135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354132, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:35:32.553653 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 14242
I20260925 16:35:32.553817 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-0/wal/instance
I20260925 16:35:32.554136 392 external_mini_cluster.cc:1428] Setting key c2f28b4495c1a9716fcb8667aaddcb89
I20260925 16:35:32.557914 14219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:32.545764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51283 (local address 127.0.98.62:35813)
0925 16:35:32.545894 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 16:35:32.545897 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:35:32.546767 (+ 870us) server_negotiation.cc:407] Connection header received
0925 16:35:32.548023 (+ 1256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:35:32.548026 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:35:32.548096 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:35:32.548184 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:35:32.549286 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:32.549805 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:32.551170 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:32.551484 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:32.551604 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:32.554104 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:35:32.554108 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:35:32.554111 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:35:32.554142 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:35:32.555864 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:32.556369 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:32.556375 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:32.556377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:32.556465 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:32.556834 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:32.556838 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:32.556840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:32.557022 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:35:32.557128 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 16:35:32.557612 (+ 484us) server_negotiation.cc:299] Negotiation successful
0925 16:35:32.557732 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 16:35:32.559012 14373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:32.546170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35813 (local address 127.0.98.1:51283)
0925 16:35:32.546620 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:32.546675 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 16:35:32.547798 (+ 1123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:35:32.548351 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:35:32.548358 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:35:32.548699 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:35:32.549142 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:32.549154 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:32.549928 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:32.549934 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:32.551055 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:32.551064 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:32.551639 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:32.551643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:32.551819 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:32.552408 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:35:32.552424 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:35:32.553936 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:35:32.555983 (+ 2047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:35:32.555989 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:35:32.556003 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:35:32.556255 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:35:32.556563 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:35:32.556566 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:35:32.556568 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:35:32.556733 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:35:32.557128 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:35:32.557135 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:35:32.557383 (+ 248us) client_negotiation.cc:769] Sending connection context
0925 16:35:32.557572 (+ 189us) client_negotiation.cc:240] Negotiation successful
0925 16:35:32.557888 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":100,"threads_started":1}
I20260925 16:35:32.560410 14371 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35813
I20260925 16:35:32.560664 14371 heartbeater.cc:464] Registering TS with master...
I20260925 16:35:32.561126 14371 heartbeater.cc:511] Master 127.0.98.62:35813 requested a full tablet report, sending...
I20260925 16:35:32.562662 14180 ts_manager.cc:194] Registered new tserver with Master: e58d56335e084712b0f4c8aecff74e8c (127.0.98.1:38835)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 16:35:32.563802 14180 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51283
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 16:35:32.581670 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35813
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:35:32.688910 14376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:35:32.689154 14376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:35:32.689193 14376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:35:32.693100 14376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:35:32.693154 14376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:35:32.693210 14376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 16:35:32.698781 14376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:35:32.699707 14376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:35:32.700603 14376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:35:32.710125 14382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:35:32.710340 14376 server_base.cc:1034] running on GCE node
W20260925 16:35:32.710197 14384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:35:32.710125 14381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:35:32.710870 14376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:35:32.711477 14376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:35:32.712633 14376 hybrid_clock.cc:648] HybridClock initialized: now 1790354132712620 us; error 26 us; skew 500 ppm
Sep 25 16:35:32 dist-test-slave-65pf krb5kdc[14135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354132, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:35:32.715344 14376 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 16:35:32.716173 14376 webserver.cc:460] Webserver started at http://127.0.98.2:33313/ using document root <none> and password file <none>
I20260925 16:35:32.716712 14376 fs_manager.cc:359] Metadata directory not provided
I20260925 16:35:32.716801 14376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:35:32.717059 14376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:35:32.719087 14376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-1/data/instance:
uuid: "2dafcd8fb7e74e919d7ec6ae0665d657"
format_stamp: "Formatted at 2026-09-25 16:35:32 on dist-test-slave-65pf"
server_key: "2c7d016aadaddec123ce70e4b2071524"
server_key_iv: "17f97ab56abe3ca691678fef7ce2bfc9"
server_key_version: "encryptionkey@0"
I20260925 16:35:32.719627 14376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2dafcd8fb7e74e919d7ec6ae0665d657"
format_stamp: "Formatted at 2026-09-25 16:35:32 on dist-test-slave-65pf"
server_key: "2c7d016aadaddec123ce70e4b2071524"
server_key_iv: "17f97ab56abe3ca691678fef7ce2bfc9"
server_key_version: "encryptionkey@0"
I20260925 16:35:32.723357 14376 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 16:35:32.725627 14391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:35:32.726831 14376 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260925 16:35:32.726951 14376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-1/wal
uuid: "2dafcd8fb7e74e919d7ec6ae0665d657"
format_stamp: "Formatted at 2026-09-25 16:35:32 on dist-test-slave-65pf"
server_key: "2c7d016aadaddec123ce70e4b2071524"
server_key_iv: "17f97ab56abe3ca691678fef7ce2bfc9"
server_key_version: "encryptionkey@0"
I20260925 16:35:32.727056 14376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:35:32.745560 14376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:35:32.748824 14376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:35:32.749020 14376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:35:32.749577 14376 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:35:32.750360 14376 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:35:32.750435 14376 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:35:32.750509 14376 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:35:32.750561 14376 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:35:32.759424 14376 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41139
I20260925 16:35:32.759449 14504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41139 every 8 connection(s)
I20260925 16:35:32.760301 14376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
I20260925 16:35:32.767700 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 14376
I20260925 16:35:32.767829 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-1/wal/instance
I20260925 16:35:32.768146 392 external_mini_cluster.cc:1428] Setting key 06572b408787f4eb09e45ace982d3f0e
Sep 25 16:35:32 dist-test-slave-65pf krb5kdc[14135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354132, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:35:32.774044 14219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:32.762108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43035 (local address 127.0.98.62:35813)
0925 16:35:32.762262 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 16:35:32.762267 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:35:32.763088 (+ 821us) server_negotiation.cc:407] Connection header received
0925 16:35:32.764290 (+ 1202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:35:32.764293 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:35:32.764347 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:35:32.764418 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:35:32.765488 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:32.766019 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:32.767098 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:32.767303 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:32.767386 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:32.770119 (+ 2733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:35:32.770123 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:35:32.770125 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:35:32.770158 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:35:32.771827 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:32.772413 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:32.772415 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:32.772417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:32.772484 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:32.772870 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:32.772872 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:32.772873 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:32.773029 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:35:32.773106 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 16:35:32.773761 (+ 655us) server_negotiation.cc:299] Negotiation successful
0925 16:35:32.773869 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 16:35:32.775096 14507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:32.762475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35813 (local address 127.0.98.2:43035)
0925 16:35:32.762940 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:32.762988 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 16:35:32.764110 (+ 1122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:35:32.764573 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:35:32.764579 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:35:32.764910 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:35:32.765372 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:32.765385 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:32.766127 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:32.766130 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:32.766992 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:32.767000 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:32.767470 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:32.767475 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:32.767671 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:32.768244 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:35:32.768263 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:35:32.769955 (+ 1692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:35:32.771940 (+ 1985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:35:32.771948 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:35:32.771963 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:35:32.772319 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:35:32.772572 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:35:32.772576 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:35:32.772577 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:35:32.772758 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:35:32.773136 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:35:32.773144 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:35:32.773447 (+ 303us) client_negotiation.cc:769] Sending connection context
0925 16:35:32.773679 (+ 232us) client_negotiation.cc:240] Negotiation successful
0925 16:35:32.774020 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":155,"threads_started":1}
I20260925 16:35:32.776322 14505 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35813
I20260925 16:35:32.776584 14505 heartbeater.cc:464] Registering TS with master...
I20260925 16:35:32.777151 14505 heartbeater.cc:511] Master 127.0.98.62:35813 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260925 16:35:32.778213 14180 ts_manager.cc:194] Registered new tserver with Master: 2dafcd8fb7e74e919d7ec6ae0665d657 (127.0.98.2:41139)
I20260925 16:35:32.778862 14180 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43035
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 16:35:32.795634 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35813
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:35:32.904393 14510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:35:32.904675 14510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:35:32.904745 14510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:35:32.909004 14510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:35:32.909091 14510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:35:32.909179 14510 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 16:35:32.914390 14510 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:35:32.915463 14510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:35:32.916486 14510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:35:32.925958 14515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:35:32.926028 14518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:35:32.926232 14510 server_base.cc:1034] running on GCE node
W20260925 16:35:32.926030 14516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:35:32.926807 14510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:35:32.927434 14510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:35:32.928599 14510 hybrid_clock.cc:648] HybridClock initialized: now 1790354132928562 us; error 49 us; skew 500 ppm
Sep 25 16:35:32 dist-test-slave-65pf krb5kdc[14135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354132, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:35:32.931368 14510 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 16:35:32.932161 14510 webserver.cc:460] Webserver started at http://127.0.98.3:44633/ using document root <none> and password file <none>
I20260925 16:35:32.932683 14510 fs_manager.cc:359] Metadata directory not provided
I20260925 16:35:32.932739 14510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:35:32.932915 14510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:35:32.934959 14510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-2/data/instance:
uuid: "efb773df56da488da71618c4eb87301a"
format_stamp: "Formatted at 2026-09-25 16:35:32 on dist-test-slave-65pf"
server_key: "a2c7077d9c105203fbce111449069d7c"
server_key_iv: "a06bf4692104a4972717f8fc2c415386"
server_key_version: "encryptionkey@0"
I20260925 16:35:32.935449 14510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-2/wal/instance:
uuid: "efb773df56da488da71618c4eb87301a"
format_stamp: "Formatted at 2026-09-25 16:35:32 on dist-test-slave-65pf"
server_key: "a2c7077d9c105203fbce111449069d7c"
server_key_iv: "a06bf4692104a4972717f8fc2c415386"
server_key_version: "encryptionkey@0"
I20260925 16:35:32.939049 14510 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 16:35:32.941399 14525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:35:32.942569 14510 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:35:32.942749 14510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-2/wal
uuid: "efb773df56da488da71618c4eb87301a"
format_stamp: "Formatted at 2026-09-25 16:35:32 on dist-test-slave-65pf"
server_key: "a2c7077d9c105203fbce111449069d7c"
server_key_iv: "a06bf4692104a4972717f8fc2c415386"
server_key_version: "encryptionkey@0"
I20260925 16:35:32.942880 14510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:35:32.969488 14510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:35:32.973055 14510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:35:32.973261 14510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:35:32.973866 14510 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:35:32.974738 14510 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:35:32.974807 14510 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:35:32.974884 14510 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:35:32.974937 14510 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:35:32.983495 14510 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34417
I20260925 16:35:32.983513 14638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34417 every 8 connection(s)
I20260925 16:35:32.984392 14510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
I20260925 16:35:32.991756 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 14510
I20260925 16:35:32.991866 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-2/wal/instance
I20260925 16:35:32.992170 392 external_mini_cluster.cc:1428] Setting key 88ed2d57b63a7829d1e43b3e632cb756
Sep 25 16:35:32 dist-test-slave-65pf krb5kdc[14135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354132, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:35:32.997475 14219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:32.986191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40349 (local address 127.0.98.62:35813)
0925 16:35:32.986329 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 16:35:32.986333 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:35:32.987217 (+ 884us) server_negotiation.cc:407] Connection header received
0925 16:35:32.988402 (+ 1185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:35:32.988405 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:35:32.988450 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:35:32.988523 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:35:32.989638 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:32.990147 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:32.991185 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:32.991387 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:32.991517 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:32.994199 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:35:32.994203 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:35:32.994206 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:35:32.994234 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:35:32.995762 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:32.996199 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:32.996202 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:32.996203 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:32.996270 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:32.996602 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:32.996604 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:32.996605 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:32.996766 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:35:32.996839 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 16:35:32.997267 (+ 428us) server_negotiation.cc:299] Negotiation successful
0925 16:35:32.997363 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 16:35:32.998288 14641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:32.986577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35813 (local address 127.0.98.3:40349)
0925 16:35:32.987065 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:32.987107 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 16:35:32.988212 (+ 1105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:35:32.988684 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:35:32.988690 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:35:32.989042 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:35:32.989502 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:32.989515 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:32.990245 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:32.990247 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:32.991087 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:32.991095 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:32.991570 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:32.991572 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:32.991712 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:32.992361 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:35:32.992376 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:35:32.994005 (+ 1629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:35:32.995857 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:35:32.995861 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:35:32.995871 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:35:32.996118 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:35:32.996376 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:35:32.996378 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:35:32.996379 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:35:32.996503 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:35:32.996849 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:35:32.996854 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:35:32.997075 (+ 221us) client_negotiation.cc:769] Sending connection context
0925 16:35:32.997225 (+ 150us) client_negotiation.cc:240] Negotiation successful
0925 16:35:32.997455 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":96,"threads_started":1}
I20260925 16:35:32.999326 14639 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35813
I20260925 16:35:32.999552 14639 heartbeater.cc:464] Registering TS with master...
I20260925 16:35:33.000010 14639 heartbeater.cc:511] Master 127.0.98.62:35813 requested a full tablet report, sending...
I20260925 16:35:33.000986 14180 ts_manager.cc:194] Registered new tserver with Master: efb773df56da488da71618c4eb87301a (127.0.98.3:34417)
I20260925 16:35:33.001569 14180 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40349
I20260925 16:35:33.005707 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 16:35:33.014163 14219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:33.007602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58930 (local address 127.0.98.62:35813)
0925 16:35:33.007740 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 16:35:33.007745 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:35:33.007899 (+ 154us) server_negotiation.cc:407] Connection header received
0925 16:35:33.007989 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:35:33.007991 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:35:33.008027 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:35:33.008084 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:35:33.008686 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:33.009161 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:33.009953 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:33.010147 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:33.010217 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:33.011546 (+ 1329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:35:33.011549 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:35:33.011550 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:35:33.011569 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:35:33.012756 (+ 1187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:33.013111 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:33.013113 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:33.013114 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:33.013179 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:33.013399 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:33.013401 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:33.013402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:33.013581 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:35:33.013671 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 16:35:33.013852 (+ 181us) server_negotiation.cc:299] Negotiation successful
0925 16:35:33.013957 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 16:35:33.017021 14180 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58930:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 16:35:33.020140 14180 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:35:33.033834 14653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:33.028157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38835 (local address 127.0.0.1:42440)
0925 16:35:33.028718 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:33.028730 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 16:35:33.028860 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:35:33.029480 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:35:33.029485 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:35:33.029507 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:35:33.029651 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:33.029658 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:33.031031 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:33.031037 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:33.032439 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:33.032449 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:33.033291 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:33.033295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:33.033401 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:33.033434 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 16:35:33.033546 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 16:35:33.033675 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"mutex_wait_us":24,"spinlock_wait_cycles":1280,"thread_start_us":93,"threads_started":1}
I20260925 16:35:33.034022 14651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:33.028078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52610 (local address 127.0.98.3:34417)
0925 16:35:33.029376 (+ 1298us) server_negotiation.cc:206] Beginning negotiation
0925 16:35:33.029390 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 16:35:33.029416 (+ 26us) server_negotiation.cc:407] Connection header received
0925 16:35:33.029485 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:35:33.029490 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:35:33.029684 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:35:33.029815 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:35:33.030083 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:33.031216 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:33.032598 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:33.033422 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:33.033534 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:33.033678 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 16:35:33.033733 (+ 55us) server_negotiation.cc:299] Negotiation successful
0925 16:35:33.033822 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1186,"thread_start_us":99,"threads_started":1}
I20260925 16:35:33.034164 14650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:33.028217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41139 (local address 127.0.0.1:58652)
0925 16:35:33.029059 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:33.029070 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 16:35:33.029134 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:35:33.029984 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:35:33.029987 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:35:33.030006 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:35:33.030089 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:33.030094 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:33.031396 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:33.031401 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:33.032595 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:33.032604 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:33.033740 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:33.033744 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:33.033841 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:33.033863 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 16:35:33.033933 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 16:35:33.034009 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":789,"thread_start_us":80,"threads_started":1}
I20260925 16:35:33.034490 14654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:33.028663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42440 (local address 127.0.98.1:38835)
0925 16:35:33.028979 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0925 16:35:33.028994 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 16:35:33.029015 (+ 21us) server_negotiation.cc:407] Connection header received
0925 16:35:33.029095 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:35:33.029103 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:35:33.029313 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:35:33.029825 (+ 512us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:35:33.030013 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:33.030909 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:33.032564 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:33.033156 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:33.033295 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:33.033440 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 16:35:33.034288 (+ 848us) server_negotiation.cc:299] Negotiation successful
0925 16:35:33.034379 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":107,"threads_started":1}
I20260925 16:35:33.033849 14652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:33.027900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34417 (local address 127.0.0.1:52610)
0925 16:35:33.028679 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:33.028728 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 16:35:33.028861 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:35:33.029821 (+ 960us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:35:33.029825 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:35:33.029841 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:35:33.029957 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:33.029963 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:33.031347 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:33.031351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:33.032500 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:33.032510 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:33.033527 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:33.033533 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:33.033613 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:33.033631 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 16:35:33.033704 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 16:35:33.033757 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"spinlock_wait_cycles":1408,"thread_start_us":84,"threads_started":1}
I20260925 16:35:33.034560 14655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:33.029043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58652 (local address 127.0.98.2:41139)
0925 16:35:33.029446 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0925 16:35:33.029459 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 16:35:33.029481 (+ 22us) server_negotiation.cc:407] Connection header received
0925 16:35:33.029557 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:35:33.029561 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:35:33.029756 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:35:33.029951 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:35:33.030176 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:33.031293 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:33.032714 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:33.033463 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:33.033548 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:33.033678 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 16:35:33.034328 (+ 650us) server_negotiation.cc:299] Negotiation successful
0925 16:35:33.034424 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":108,"threads_started":1}
I20260925 16:35:33.037106 14573 tablet_service.cc:1467] Processing CreateTablet for tablet f4467bd68d5d461880f7a500d8215c6c (DEFAULT_TABLE table=test-table [id=3298dde6786e4b84bc8c127ea989fb38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:35:33.037223 14439 tablet_service.cc:1467] Processing CreateTablet for tablet f4467bd68d5d461880f7a500d8215c6c (DEFAULT_TABLE table=test-table [id=3298dde6786e4b84bc8c127ea989fb38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:35:33.037223 14305 tablet_service.cc:1467] Processing CreateTablet for tablet f4467bd68d5d461880f7a500d8215c6c (DEFAULT_TABLE table=test-table [id=3298dde6786e4b84bc8c127ea989fb38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:35:33.038139 14573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4467bd68d5d461880f7a500d8215c6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:35:33.038187 14305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4467bd68d5d461880f7a500d8215c6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:35:33.038187 14439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4467bd68d5d461880f7a500d8215c6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:35:33.044533 14656 tablet_bootstrap.cc:492] T f4467bd68d5d461880f7a500d8215c6c P efb773df56da488da71618c4eb87301a: Bootstrap starting.
I20260925 16:35:33.045779 14658 tablet_bootstrap.cc:492] T f4467bd68d5d461880f7a500d8215c6c P 2dafcd8fb7e74e919d7ec6ae0665d657: Bootstrap starting.
I20260925 16:35:33.045989 14657 tablet_bootstrap.cc:492] T f4467bd68d5d461880f7a500d8215c6c P e58d56335e084712b0f4c8aecff74e8c: Bootstrap starting.
I20260925 16:35:33.046873 14656 tablet_bootstrap.cc:654] T f4467bd68d5d461880f7a500d8215c6c P efb773df56da488da71618c4eb87301a: Neither blocks nor log segments found. Creating new log.
I20260925 16:35:33.047895 14657 tablet_bootstrap.cc:654] T f4467bd68d5d461880f7a500d8215c6c P e58d56335e084712b0f4c8aecff74e8c: Neither blocks nor log segments found. Creating new log.
I20260925 16:35:33.048038 14656 log.cc:824] T f4467bd68d5d461880f7a500d8215c6c P efb773df56da488da71618c4eb87301a: Log is configured to *not* fsync() on all Append() calls
I20260925 16:35:33.048306 14658 tablet_bootstrap.cc:654] T f4467bd68d5d461880f7a500d8215c6c P 2dafcd8fb7e74e919d7ec6ae0665d657: Neither blocks nor log segments found. Creating new log.
I20260925 16:35:33.049113 14658 log.cc:824] T f4467bd68d5d461880f7a500d8215c6c P 2dafcd8fb7e74e919d7ec6ae0665d657: Log is configured to *not* fsync() on all Append() calls
I20260925 16:35:33.049113 14657 log.cc:824] T f4467bd68d5d461880f7a500d8215c6c P e58d56335e084712b0f4c8aecff74e8c: Log is configured to *not* fsync() on all Append() calls
I20260925 16:35:33.050549 14656 tablet_bootstrap.cc:492] T f4467bd68d5d461880f7a500d8215c6c P efb773df56da488da71618c4eb87301a: No bootstrap required, opened a new log
I20260925 16:35:33.050786 14657 tablet_bootstrap.cc:492] T f4467bd68d5d461880f7a500d8215c6c P e58d56335e084712b0f4c8aecff74e8c: No bootstrap required, opened a new log
I20260925 16:35:33.050839 14656 ts_tablet_manager.cc:1397] T f4467bd68d5d461880f7a500d8215c6c P efb773df56da488da71618c4eb87301a: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 16:35:33.050980 14657 ts_tablet_manager.cc:1397] T f4467bd68d5d461880f7a500d8215c6c P e58d56335e084712b0f4c8aecff74e8c: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.004s
I20260925 16:35:33.051111 14658 tablet_bootstrap.cc:492] T f4467bd68d5d461880f7a500d8215c6c P 2dafcd8fb7e74e919d7ec6ae0665d657: No bootstrap required, opened a new log
I20260925 16:35:33.051288 14658 ts_tablet_manager.cc:1397] T f4467bd68d5d461880f7a500d8215c6c P 2dafcd8fb7e74e919d7ec6ae0665d657: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 16:35:33.053879 14657 raft_consensus.cc:348] T f4467bd68d5d461880f7a500d8215c6c P e58d56335e084712b0f4c8aecff74e8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb773df56da488da71618c4eb87301a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34417 } } peers { permanent_uuid: "e58d56335e084712b0f4c8aecff74e8c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38835 } } peers { permanent_uuid: "2dafcd8fb7e74e919d7ec6ae0665d657" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41139 } }
I20260925 16:35:33.054116 14657 raft_consensus.cc:374] T f4467bd68d5d461880f7a500d8215c6c P e58d56335e084712b0f4c8aecff74e8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:35:33.054175 14657 raft_consensus.cc:729] T f4467bd68d5d461880f7a500d8215c6c P e58d56335e084712b0f4c8aecff74e8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e58d56335e084712b0f4c8aecff74e8c, State: Initialized, Role: FOLLOWER
I20260925 16:35:33.054689 14657 consensus_queue.cc:260] T f4467bd68d5d461880f7a500d8215c6c P e58d56335e084712b0f4c8aecff74e8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb773df56da488da71618c4eb87301a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34417 } } peers { permanent_uuid: "e58d56335e084712b0f4c8aecff74e8c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38835 } } peers { permanent_uuid: "2dafcd8fb7e74e919d7ec6ae0665d657" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41139 } }
I20260925 16:35:33.054901 14656 raft_consensus.cc:348] T f4467bd68d5d461880f7a500d8215c6c P efb773df56da488da71618c4eb87301a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb773df56da488da71618c4eb87301a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34417 } } peers { permanent_uuid: "e58d56335e084712b0f4c8aecff74e8c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38835 } } peers { permanent_uuid: "2dafcd8fb7e74e919d7ec6ae0665d657" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41139 } }
I20260925 16:35:33.054914 14658 raft_consensus.cc:348] T f4467bd68d5d461880f7a500d8215c6c P 2dafcd8fb7e74e919d7ec6ae0665d657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb773df56da488da71618c4eb87301a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34417 } } peers { permanent_uuid: "e58d56335e084712b0f4c8aecff74e8c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38835 } } peers { permanent_uuid: "2dafcd8fb7e74e919d7ec6ae0665d657" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41139 } }
I20260925 16:35:33.055166 14658 raft_consensus.cc:374] T f4467bd68d5d461880f7a500d8215c6c P 2dafcd8fb7e74e919d7ec6ae0665d657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:35:33.055248 14656 raft_consensus.cc:374] T f4467bd68d5d461880f7a500d8215c6c P efb773df56da488da71618c4eb87301a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:35:33.055253 14658 raft_consensus.cc:729] T f4467bd68d5d461880f7a500d8215c6c P 2dafcd8fb7e74e919d7ec6ae0665d657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dafcd8fb7e74e919d7ec6ae0665d657, State: Initialized, Role: FOLLOWER
I20260925 16:35:33.055336 14371 heartbeater.cc:503] Master 127.0.98.62:35813 was elected leader, sending a full tablet report...
I20260925 16:35:33.055356 14656 raft_consensus.cc:729] T f4467bd68d5d461880f7a500d8215c6c P efb773df56da488da71618c4eb87301a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efb773df56da488da71618c4eb87301a, State: Initialized, Role: FOLLOWER
I20260925 16:35:33.055691 14657 ts_tablet_manager.cc:1428] T f4467bd68d5d461880f7a500d8215c6c P e58d56335e084712b0f4c8aecff74e8c: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260925 16:35:33.055675 14658 consensus_queue.cc:260] T f4467bd68d5d461880f7a500d8215c6c P 2dafcd8fb7e74e919d7ec6ae0665d657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb773df56da488da71618c4eb87301a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34417 } } peers { permanent_uuid: "e58d56335e084712b0f4c8aecff74e8c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38835 } } peers { permanent_uuid: "2dafcd8fb7e74e919d7ec6ae0665d657" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41139 } }
I20260925 16:35:33.056354 14505 heartbeater.cc:503] Master 127.0.98.62:35813 was elected leader, sending a full tablet report...
I20260925 16:35:33.056607 14658 ts_tablet_manager.cc:1428] T f4467bd68d5d461880f7a500d8215c6c P 2dafcd8fb7e74e919d7ec6ae0665d657: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 16:35:33.056656 14656 consensus_queue.cc:260] T f4467bd68d5d461880f7a500d8215c6c P efb773df56da488da71618c4eb87301a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb773df56da488da71618c4eb87301a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34417 } } peers { permanent_uuid: "e58d56335e084712b0f4c8aecff74e8c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38835 } } peers { permanent_uuid: "2dafcd8fb7e74e919d7ec6ae0665d657" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41139 } }
I20260925 16:35:33.057307 14639 heartbeater.cc:503] Master 127.0.98.62:35813 was elected leader, sending a full tablet report...
I20260925 16:35:33.058383 14656 ts_tablet_manager.cc:1428] T f4467bd68d5d461880f7a500d8215c6c P efb773df56da488da71618c4eb87301a: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260925 16:35:33.198181 14663 raft_consensus.cc:482] T f4467bd68d5d461880f7a500d8215c6c P 2dafcd8fb7e74e919d7ec6ae0665d657 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:35:33.198395 14663 raft_consensus.cc:504] T f4467bd68d5d461880f7a500d8215c6c P 2dafcd8fb7e74e919d7ec6ae0665d657 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb773df56da488da71618c4eb87301a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34417 } } peers { permanent_uuid: "e58d56335e084712b0f4c8aecff74e8c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38835 } } peers { permanent_uuid: "2dafcd8fb7e74e919d7ec6ae0665d657" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41139 } }
I20260925 16:35:33.199564 14663 leader_election.cc:288] T f4467bd68d5d461880f7a500d8215c6c P 2dafcd8fb7e74e919d7ec6ae0665d657 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efb773df56da488da71618c4eb87301a (127.0.98.3:34417), e58d56335e084712b0f4c8aecff74e8c (127.0.98.1:38835)
I20260925 16:35:33.203711 14665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:33.199923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38835 (local address 127.0.98.2:47239)
0925 16:35:33.200295 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:33.200307 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 16:35:33.200395 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:35:33.200706 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:35:33.200708 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:35:33.200724 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:35:33.200860 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:33.200866 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:33.201729 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:33.201732 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:33.202530 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:33.202539 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:33.203300 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:33.203303 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:33.203382 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:33.203399 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 16:35:33.203491 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 16:35:33.203557 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"mutex_wait_us":42,"thread_start_us":122,"threads_started":1}
I20260925 16:35:33.203696 14651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:33.199928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51327 (local address 127.0.98.3:34417)
0925 16:35:33.200072 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 16:35:33.200077 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:35:33.200095 (+ 18us) server_negotiation.cc:407] Connection header received
0925 16:35:33.200152 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:35:33.200157 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:35:33.200207 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:35:33.200286 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:35:33.200710 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:33.201309 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:33.202634 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:33.203063 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:33.203145 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:33.203194 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 16:35:33.203511 (+ 317us) server_negotiation.cc:299] Negotiation successful
0925 16:35:33.203574 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 16:35:33.203730 14654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:33.200114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47239 (local address 127.0.98.1:38835)
0925 16:35:33.200292 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 16:35:33.200301 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 16:35:33.200388 (+ 87us) server_negotiation.cc:407] Connection header received
0925 16:35:33.200511 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:35:33.200517 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:35:33.200576 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:35:33.200683 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:35:33.200980 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:33.201608 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:33.202692 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:33.203174 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:33.203304 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:33.203378 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 16:35:33.203510 (+ 132us) server_negotiation.cc:299] Negotiation successful
0925 16:35:33.203582 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 16:35:33.203730 14507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:33.199765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34417 (local address 127.0.98.2:51327)
0925 16:35:33.199918 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:33.199936 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 16:35:33.200032 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:35:33.200373 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:35:33.200381 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:35:33.200411 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:35:33.200566 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:33.200575 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:33.201434 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:33.201438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:33.202481 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:33.202491 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:33.203188 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:33.203193 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:33.203341 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:33.203367 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 16:35:33.203498 (+ 131us) client_negotiation.cc:240] Negotiation successful
0925 16:35:33.203582 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":1}
I20260925 16:35:33.204558 14593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4467bd68d5d461880f7a500d8215c6c" candidate_uuid: "2dafcd8fb7e74e919d7ec6ae0665d657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efb773df56da488da71618c4eb87301a" is_pre_election: true
I20260925 16:35:33.204562 14325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4467bd68d5d461880f7a500d8215c6c" candidate_uuid: "2dafcd8fb7e74e919d7ec6ae0665d657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e58d56335e084712b0f4c8aecff74e8c" is_pre_election: true
I20260925 16:35:33.204873 14593 raft_consensus.cc:2454] T f4467bd68d5d461880f7a500d8215c6c P efb773df56da488da71618c4eb87301a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2dafcd8fb7e74e919d7ec6ae0665d657 in term 0.
I20260925 16:35:33.204877 14325 raft_consensus.cc:2454] T f4467bd68d5d461880f7a500d8215c6c P e58d56335e084712b0f4c8aecff74e8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2dafcd8fb7e74e919d7ec6ae0665d657 in term 0.
I20260925 16:35:33.205350 14394 leader_election.cc:302] T f4467bd68d5d461880f7a500d8215c6c P 2dafcd8fb7e74e919d7ec6ae0665d657 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dafcd8fb7e74e919d7ec6ae0665d657, e58d56335e084712b0f4c8aecff74e8c; no voters:
I20260925 16:35:33.205644 14663 raft_consensus.cc:2781] T f4467bd68d5d461880f7a500d8215c6c P 2dafcd8fb7e74e919d7ec6ae0665d657 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:35:33.205739 14663 raft_consensus.cc:482] T f4467bd68d5d461880f7a500d8215c6c P 2dafcd8fb7e74e919d7ec6ae0665d657 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:35:33.205816 14663 raft_consensus.cc:3037] T f4467bd68d5d461880f7a500d8215c6c P 2dafcd8fb7e74e919d7ec6ae0665d657 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:35:33.207121 14663 raft_consensus.cc:504] T f4467bd68d5d461880f7a500d8215c6c P 2dafcd8fb7e74e919d7ec6ae0665d657 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb773df56da488da71618c4eb87301a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34417 } } peers { permanent_uuid: "e58d56335e084712b0f4c8aecff74e8c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38835 } } peers { permanent_uuid: "2dafcd8fb7e74e919d7ec6ae0665d657" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41139 } }
I20260925 16:35:33.207576 14663 leader_election.cc:288] T f4467bd68d5d461880f7a500d8215c6c P 2dafcd8fb7e74e919d7ec6ae0665d657 [CANDIDATE]: Term 1 election: Requested vote from peers efb773df56da488da71618c4eb87301a (127.0.98.3:34417), e58d56335e084712b0f4c8aecff74e8c (127.0.98.1:38835)
I20260925 16:35:33.208087 14593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4467bd68d5d461880f7a500d8215c6c" candidate_uuid: "2dafcd8fb7e74e919d7ec6ae0665d657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efb773df56da488da71618c4eb87301a"
I20260925 16:35:33.208211 14325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4467bd68d5d461880f7a500d8215c6c" candidate_uuid: "2dafcd8fb7e74e919d7ec6ae0665d657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e58d56335e084712b0f4c8aecff74e8c"
I20260925 16:35:33.208240 14593 raft_consensus.cc:3037] T f4467bd68d5d461880f7a500d8215c6c P efb773df56da488da71618c4eb87301a [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:35:33.208334 14325 raft_consensus.cc:3037] T f4467bd68d5d461880f7a500d8215c6c P e58d56335e084712b0f4c8aecff74e8c [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:35:33.209527 14593 raft_consensus.cc:2454] T f4467bd68d5d461880f7a500d8215c6c P efb773df56da488da71618c4eb87301a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2dafcd8fb7e74e919d7ec6ae0665d657 in term 1.
I20260925 16:35:33.209527 14325 raft_consensus.cc:2454] T f4467bd68d5d461880f7a500d8215c6c P e58d56335e084712b0f4c8aecff74e8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2dafcd8fb7e74e919d7ec6ae0665d657 in term 1.
I20260925 16:35:33.209827 14393 leader_election.cc:302] T f4467bd68d5d461880f7a500d8215c6c P 2dafcd8fb7e74e919d7ec6ae0665d657 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dafcd8fb7e74e919d7ec6ae0665d657, efb773df56da488da71618c4eb87301a; no voters:
I20260925 16:35:33.210008 14663 raft_consensus.cc:2781] T f4467bd68d5d461880f7a500d8215c6c P 2dafcd8fb7e74e919d7ec6ae0665d657 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:35:33.210258 14663 raft_consensus.cc:686] T f4467bd68d5d461880f7a500d8215c6c P 2dafcd8fb7e74e919d7ec6ae0665d657 [term 1 LEADER]: Becoming Leader. State: Replica: 2dafcd8fb7e74e919d7ec6ae0665d657, State: Running, Role: LEADER
I20260925 16:35:33.210608 14663 consensus_queue.cc:237] T f4467bd68d5d461880f7a500d8215c6c P 2dafcd8fb7e74e919d7ec6ae0665d657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb773df56da488da71618c4eb87301a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34417 } } peers { permanent_uuid: "e58d56335e084712b0f4c8aecff74e8c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38835 } } peers { permanent_uuid: "2dafcd8fb7e74e919d7ec6ae0665d657" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41139 } }
I20260925 16:35:33.213768 14180 catalog_manager.cc:5495] T f4467bd68d5d461880f7a500d8215c6c P 2dafcd8fb7e74e919d7ec6ae0665d657 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2dafcd8fb7e74e919d7ec6ae0665d657 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "2dafcd8fb7e74e919d7ec6ae0665d657" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb773df56da488da71618c4eb87301a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e58d56335e084712b0f4c8aecff74e8c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2dafcd8fb7e74e919d7ec6ae0665d657" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41139 } health_report { overall_health: HEALTHY } } }
W20260925 16:35:33.235280 14640 tablet.cc:2369] T f4467bd68d5d461880f7a500d8215c6c P efb773df56da488da71618c4eb87301a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 16:35:33.261461 14506 tablet.cc:2369] T f4467bd68d5d461880f7a500d8215c6c P 2dafcd8fb7e74e919d7ec6ae0665d657: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:35:33.267700 14655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:33.264192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58666 (local address 127.0.98.2:41139)
0925 16:35:33.264334 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 16:35:33.264339 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:35:33.264353 (+ 14us) server_negotiation.cc:407] Connection header received
0925 16:35:33.264454 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:35:33.264457 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:35:33.264519 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:35:33.264632 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:35:33.264873 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:33.265375 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:33.266244 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:33.266468 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:33.266581 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:33.266893 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:35:33.267303 (+ 410us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:35:33.267436 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 16:35:33.267488 (+ 52us) server_negotiation.cc:299] Negotiation successful
0925 16:35:33.267542 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 16:35:33.273852 14593 raft_consensus.cc:1261] T f4467bd68d5d461880f7a500d8215c6c P efb773df56da488da71618c4eb87301a [term 1 FOLLOWER]: Refusing update from remote peer 2dafcd8fb7e74e919d7ec6ae0665d657: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:35:33.274506 14325 raft_consensus.cc:1261] T f4467bd68d5d461880f7a500d8215c6c P e58d56335e084712b0f4c8aecff74e8c [term 1 FOLLOWER]: Refusing update from remote peer 2dafcd8fb7e74e919d7ec6ae0665d657: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:35:33.274546 14663 consensus_queue.cc:1035] T f4467bd68d5d461880f7a500d8215c6c P 2dafcd8fb7e74e919d7ec6ae0665d657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb773df56da488da71618c4eb87301a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:35:33.275132 14663 consensus_queue.cc:1035] T f4467bd68d5d461880f7a500d8215c6c P 2dafcd8fb7e74e919d7ec6ae0665d657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e58d56335e084712b0f4c8aecff74e8c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:35:33.281576 14669 mvcc.cc:205] Tried to move back new op lower bound from 7333290529882169344 to 7333290529633472512. Current Snapshot: MvccSnapshot[applied={T|T < 7333290529882169344}]
I20260925 16:35:33.284440 14670 mvcc.cc:205] Tried to move back new op lower bound from 7333290529882169344 to 7333290529633472512. Current Snapshot: MvccSnapshot[applied={T|T < 7333290529882169344}]
I20260925 16:35:33.284448 14671 mvcc.cc:205] Tried to move back new op lower bound from 7333290529882169344 to 7333290529633472512. Current Snapshot: MvccSnapshot[applied={T|T < 7333290529882169344}]
I20260925 16:35:33.291409 14180 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:35:33.291595 14180 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:35:33.294062 14180 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0b6a310d97744021b7dd800019cc5eef: Sending DeleteTablet for 3 replicas of tablet f4467bd68d5d461880f7a500d8215c6c
I20260925 16:35:33.294849 14573 tablet_service.cc:1514] Processing DeleteTablet for tablet f4467bd68d5d461880f7a500d8215c6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:35:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52610
I20260925 16:35:33.294926 14305 tablet_service.cc:1514] Processing DeleteTablet for tablet f4467bd68d5d461880f7a500d8215c6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:35:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42440
I20260925 16:35:33.295410 14678 tablet_replica.cc:330] T f4467bd68d5d461880f7a500d8215c6c P efb773df56da488da71618c4eb87301a: stopping tablet replica
I20260925 16:35:33.295435 14439 tablet_service.cc:1514] Processing DeleteTablet for tablet f4467bd68d5d461880f7a500d8215c6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:35:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58652
I20260925 16:35:33.295764 14678 raft_consensus.cc:2229] T f4467bd68d5d461880f7a500d8215c6c P efb773df56da488da71618c4eb87301a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:35:33.295951 14680 tablet_replica.cc:330] T f4467bd68d5d461880f7a500d8215c6c P 2dafcd8fb7e74e919d7ec6ae0665d657: stopping tablet replica
I20260925 16:35:33.296059 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 14242
I20260925 16:35:33.296090 14678 raft_consensus.cc:2258] T f4467bd68d5d461880f7a500d8215c6c P efb773df56da488da71618c4eb87301a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:35:33.297295 14680 raft_consensus.cc:2229] T f4467bd68d5d461880f7a500d8215c6c P 2dafcd8fb7e74e919d7ec6ae0665d657 [term 1 LEADER]: Raft consensus shutting down.
I20260925 16:35:33.297662 14680 raft_consensus.cc:2258] T f4467bd68d5d461880f7a500d8215c6c P 2dafcd8fb7e74e919d7ec6ae0665d657 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:35:33.299172 14680 ts_tablet_manager.cc:1905] T f4467bd68d5d461880f7a500d8215c6c P 2dafcd8fb7e74e919d7ec6ae0665d657: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:35:33.299795 14678 ts_tablet_manager.cc:1905] T f4467bd68d5d461880f7a500d8215c6c P efb773df56da488da71618c4eb87301a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:35:33.301884 14680 ts_tablet_manager.cc:1918] T f4467bd68d5d461880f7a500d8215c6c P 2dafcd8fb7e74e919d7ec6ae0665d657: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:35:33.302055 14680 log.cc:1196] T f4467bd68d5d461880f7a500d8215c6c P 2dafcd8fb7e74e919d7ec6ae0665d657: Deleting WAL directory at /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-1/wal/wals/f4467bd68d5d461880f7a500d8215c6c
I20260925 16:35:33.302420 14680 ts_tablet_manager.cc:1939] T f4467bd68d5d461880f7a500d8215c6c P 2dafcd8fb7e74e919d7ec6ae0665d657: Deleting consensus metadata
I20260925 16:35:33.302500 14678 ts_tablet_manager.cc:1918] T f4467bd68d5d461880f7a500d8215c6c P efb773df56da488da71618c4eb87301a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:35:33.302615 14678 log.cc:1196] T f4467bd68d5d461880f7a500d8215c6c P efb773df56da488da71618c4eb87301a: Deleting WAL directory at /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-2/wal/wals/f4467bd68d5d461880f7a500d8215c6c
W20260925 16:35:33.302953 14166 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38835 (error 108)
I20260925 16:35:33.302994 14678 ts_tablet_manager.cc:1939] T f4467bd68d5d461880f7a500d8215c6c P efb773df56da488da71618c4eb87301a: Deleting consensus metadata
I20260925 16:35:33.303751 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 14376
I20260925 16:35:33.303884 14164 catalog_manager.cc:4841] TS 2dafcd8fb7e74e919d7ec6ae0665d657 (127.0.98.2:41139): tablet f4467bd68d5d461880f7a500d8215c6c (table test-table [id=3298dde6786e4b84bc8c127ea989fb38]) successfully deleted
I20260925 16:35:33.304944 14165 catalog_manager.cc:4841] TS efb773df56da488da71618c4eb87301a (127.0.98.3:34417): tablet f4467bd68d5d461880f7a500d8215c6c (table test-table [id=3298dde6786e4b84bc8c127ea989fb38]) successfully deleted
I20260925 16:35:33.305387 14652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:33.305010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38835 (local address 127.0.0.1:42450)
0925 16:35:33.305212 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:33.305287 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
W20260925 16:35:33.305577 14166 catalog_manager.cc:4584] TS e58d56335e084712b0f4c8aecff74e8c (127.0.98.1:38835): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f4467bd68d5d461880f7a500d8215c6c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38835: connect: Connection refused (error 111)
I20260925 16:35:33.309849 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 14510
I20260925 16:35:33.315281 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 14148
I20260925 16:35:33.321390 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35813
--webserver_interface=127.0.98.62
--webserver_port=34251
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35813
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:35:33.433070 14682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:35:33.433450 14682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:35:33.433526 14682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:35:33.437654 14682 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:35:33.437726 14682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:35:33.437745 14682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:35:33.437762 14682 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:35:33.437777 14682 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:35:33.443253 14682 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=34251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:35:33.444363 14682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:35:33.445405 14682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:35:33.458282 14689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:35:33.458485 14682 server_base.cc:1034] running on GCE node
W20260925 16:35:33.458282 14691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:35:33.458284 14688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:35:33.460047 14682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:35:33.461239 14682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:35:33.462433 14682 hybrid_clock.cc:648] HybridClock initialized: now 1790354133462419 us; error 35 us; skew 500 ppm
Sep 25 16:35:33 dist-test-slave-65pf krb5kdc[14135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354133, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:35:33.465337 14682 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 16:35:33.466346 14682 webserver.cc:460] Webserver started at http://127.0.98.62:34251/ using document root <none> and password file <none>
I20260925 16:35:33.467018 14682 fs_manager.cc:359] Metadata directory not provided
I20260925 16:35:33.467128 14682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:35:33.471140 14682 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 16:35:33.473381 14698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:35:33.474736 14682 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:35:33.474884 14682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/master-0/wal
uuid: "0b6a310d97744021b7dd800019cc5eef"
format_stamp: "Formatted at 2026-09-25 16:35:32 on dist-test-slave-65pf"
server_key: "47e02e14eb355f1041a158b1faf73cc9"
server_key_iv: "1ea01d8ead1e8ff16019783038effe46"
server_key_version: "encryptionkey@0"
I20260925 16:35:33.475497 14682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:35:33.508970 14682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:35:33.512454 14682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:35:33.512669 14682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:35:33.520009 14682 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35813
I20260925 16:35:33.520308 14750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35813 every 8 connection(s)
I20260925 16:35:33.521154 14682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:35:33.525454 14751 sys_catalog.cc:263] Verifying existing consensus state
I20260925 16:35:33.527220 14751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b6a310d97744021b7dd800019cc5eef: Bootstrap starting.
I20260925 16:35:33.527568 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 14682
I20260925 16:35:33.527982 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:38835
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:35813
--webserver_port=33983
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 16:35:33.544137 14751 log.cc:824] T 00000000000000000000000000000000 P 0b6a310d97744021b7dd800019cc5eef: Log is configured to *not* fsync() on all Append() calls
I20260925 16:35:33.552822 14751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b6a310d97744021b7dd800019cc5eef: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 16:35:33.553222 14751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b6a310d97744021b7dd800019cc5eef: Bootstrap complete.
I20260925 16:35:33.557264 14751 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b6a310d97744021b7dd800019cc5eef [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b6a310d97744021b7dd800019cc5eef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35813 } }
I20260925 16:35:33.557679 14751 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b6a310d97744021b7dd800019cc5eef [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6a310d97744021b7dd800019cc5eef, State: Initialized, Role: FOLLOWER
I20260925 16:35:33.558164 14751 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b6a310d97744021b7dd800019cc5eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b6a310d97744021b7dd800019cc5eef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35813 } }
I20260925 16:35:33.558324 14751 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b6a310d97744021b7dd800019cc5eef [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:35:33.558409 14751 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b6a310d97744021b7dd800019cc5eef [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:35:33.558542 14751 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b6a310d97744021b7dd800019cc5eef [term 1 FOLLOWER]: Advancing to term 2
I20260925 16:35:33.560573 14751 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b6a310d97744021b7dd800019cc5eef [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b6a310d97744021b7dd800019cc5eef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35813 } }
I20260925 16:35:33.560925 14751 leader_election.cc:302] T 00000000000000000000000000000000 P 0b6a310d97744021b7dd800019cc5eef [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b6a310d97744021b7dd800019cc5eef; no voters:
I20260925 16:35:33.561234 14751 leader_election.cc:288] T 00000000000000000000000000000000 P 0b6a310d97744021b7dd800019cc5eef [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 16:35:33.561383 14755 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b6a310d97744021b7dd800019cc5eef [term 2 FOLLOWER]: Leader election won for term 2
I20260925 16:35:33.561635 14755 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b6a310d97744021b7dd800019cc5eef [term 2 LEADER]: Becoming Leader. State: Replica: 0b6a310d97744021b7dd800019cc5eef, State: Running, Role: LEADER
I20260925 16:35:33.562034 14755 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b6a310d97744021b7dd800019cc5eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b6a310d97744021b7dd800019cc5eef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35813 } }
I20260925 16:35:33.562428 14751 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b6a310d97744021b7dd800019cc5eef [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:35:33.563387 14756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b6a310d97744021b7dd800019cc5eef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0b6a310d97744021b7dd800019cc5eef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b6a310d97744021b7dd800019cc5eef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35813 } } }
I20260925 16:35:33.563521 14756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b6a310d97744021b7dd800019cc5eef [sys.catalog]: This master's current role is: LEADER
I20260925 16:35:33.563814 14757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b6a310d97744021b7dd800019cc5eef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b6a310d97744021b7dd800019cc5eef. Latest consensus state: current_term: 2 leader_uuid: "0b6a310d97744021b7dd800019cc5eef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b6a310d97744021b7dd800019cc5eef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35813 } } }
I20260925 16:35:33.563895 14757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b6a310d97744021b7dd800019cc5eef [sys.catalog]: This master's current role is: LEADER
I20260925 16:35:33.564333 14764 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:35:33.568964 14764 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:35:33.569531 14764 catalog_manager.cc:1259] Loaded cluster ID: 6adea33195954dbeb67088ce5172c8ed
I20260925 16:35:33.569589 14764 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:35:33.570883 14764 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260925 16:35:33.575572 14764 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260925 16:35:33.581032 14764 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:35:33.581755 14764 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0b6a310d97744021b7dd800019cc5eef: Loaded TSK: 0
I20260925 16:35:33.582975 14764 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 16:35:33.644976 14753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:35:33.645323 14753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:35:33.645380 14753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:35:33.650909 14753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:35:33.650985 14753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:35:33.651070 14753 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 16:35:33.657749 14753 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:38835
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=33983
--tserver_master_addrs=127.0.98.62:35813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14753
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:35:33.658913 14753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:35:33.659986 14753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:35:33.669798 14778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:35:33.669847 14779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:35:33.670065 14753 server_base.cc:1034] running on GCE node
W20260925 16:35:33.669798 14781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:35:33.670663 14753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:35:33.671491 14753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:35:33.672668 14753 hybrid_clock.cc:648] HybridClock initialized: now 1790354133672643 us; error 49 us; skew 500 ppm
Sep 25 16:35:33 dist-test-slave-65pf krb5kdc[14135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354133, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:35:33.675557 14753 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 16:35:33.676501 14753 webserver.cc:460] Webserver started at http://127.0.98.1:33983/ using document root <none> and password file <none>
I20260925 16:35:33.677052 14753 fs_manager.cc:359] Metadata directory not provided
I20260925 16:35:33.677160 14753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:35:33.681411 14753 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 16:35:33.683769 14788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:35:33.685109 14753 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 16:35:33.685204 14753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-0/wal
uuid: "e58d56335e084712b0f4c8aecff74e8c"
format_stamp: "Formatted at 2026-09-25 16:35:32 on dist-test-slave-65pf"
server_key: "e8d8a16ebfeb835b45e1ac4d80f7e1a3"
server_key_iv: "4ac91e6f2b8d6148b67d33911daf5d94"
server_key_version: "encryptionkey@0"
I20260925 16:35:33.685792 14753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:35:33.713161 14753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:35:33.716624 14753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:35:33.716825 14753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:35:33.717494 14753 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:35:33.718653 14795 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 16:35:33.722616 14753 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 16:35:33.722682 14753 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260925 16:35:33.722751 14753 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 16:35:33.724391 14753 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 16:35:33.724448 14753 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260925 16:35:33.724704 14795 tablet_bootstrap.cc:492] T f4467bd68d5d461880f7a500d8215c6c P e58d56335e084712b0f4c8aecff74e8c: Bootstrap starting.
I20260925 16:35:33.735140 14753 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38835
I20260925 16:35:33.735481 14902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38835 every 8 connection(s)
I20260925 16:35:33.736366 14753 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
I20260925 16:35:33.744772 14795 log.cc:824] T f4467bd68d5d461880f7a500d8215c6c P e58d56335e084712b0f4c8aecff74e8c: Log is configured to *not* fsync() on all Append() calls
I20260925 16:35:33.745460 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 14753
I20260925 16:35:33.745983 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:41139
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:35813
--webserver_port=33313
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 16:35:33 dist-test-slave-65pf krb5kdc[14135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354133, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:35:33.754655 14795 tablet_bootstrap.cc:492] T f4467bd68d5d461880f7a500d8215c6c P e58d56335e084712b0f4c8aecff74e8c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 16:35:33.755141 14795 tablet_bootstrap.cc:492] T f4467bd68d5d461880f7a500d8215c6c P e58d56335e084712b0f4c8aecff74e8c: Bootstrap complete.
I20260925 16:35:33.755548 14905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:33.738720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45253 (local address 127.0.98.62:35813)
0925 16:35:33.739146 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0925 16:35:33.739170 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0925 16:35:33.739541 (+ 371us) server_negotiation.cc:407] Connection header received
0925 16:35:33.740775 (+ 1234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:35:33.740794 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:35:33.741112 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:35:33.741516 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:35:33.742960 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:33.744014 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:33.745574 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:33.746068 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:33.746250 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:33.749474 (+ 3224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:35:33.749480 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:35:33.749493 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:35:33.749526 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:35:33.751693 (+ 2167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:33.752382 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:33.752388 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:33.752392 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:33.752527 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:33.752956 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:33.752966 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:33.752968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:33.753479 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:35:33.753650 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 16:35:33.754259 (+ 609us) server_negotiation.cc:299] Negotiation successful
0925 16:35:33.754548 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":162,"threads_started":1}
I20260925 16:35:33.755568 14906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:33.738695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35813 (local address 127.0.98.1:45253)
0925 16:35:33.739359 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:33.739405 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 16:35:33.740315 (+ 910us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:35:33.741576 (+ 1261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:35:33.741585 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:35:33.742135 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:35:33.742761 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:33.742780 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:33.744222 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:33.744225 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:33.745393 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:33.745409 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:33.746417 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:33.746426 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:33.746723 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:33.747487 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:35:33.747518 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:35:33.749234 (+ 1716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:35:33.751872 (+ 2638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:35:33.751880 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:35:33.751894 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:35:33.752204 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:35:33.752660 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:35:33.752664 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:35:33.752666 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:35:33.752829 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:35:33.753625 (+ 796us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:35:33.753632 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:35:33.753913 (+ 281us) client_negotiation.cc:769] Sending connection context
0925 16:35:33.754117 (+ 204us) client_negotiation.cc:240] Negotiation successful
0925 16:35:33.754511 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":181,"threads_started":1}
I20260925 16:35:33.756675 14795 ts_tablet_manager.cc:1397] T f4467bd68d5d461880f7a500d8215c6c P e58d56335e084712b0f4c8aecff74e8c: Time spent bootstrapping tablet: real 0.032s user 0.021s sys 0.008s
I20260925 16:35:33.758651 14903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35813
I20260925 16:35:33.758905 14903 heartbeater.cc:464] Registering TS with master...
I20260925 16:35:33.759438 14903 heartbeater.cc:511] Master 127.0.98.62:35813 requested a full tablet report, sending...
I20260925 16:35:33.759943 14795 raft_consensus.cc:348] T f4467bd68d5d461880f7a500d8215c6c P e58d56335e084712b0f4c8aecff74e8c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb773df56da488da71618c4eb87301a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34417 } } peers { permanent_uuid: "e58d56335e084712b0f4c8aecff74e8c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38835 } } peers { permanent_uuid: "2dafcd8fb7e74e919d7ec6ae0665d657" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41139 } }
I20260925 16:35:33.760460 14795 raft_consensus.cc:729] T f4467bd68d5d461880f7a500d8215c6c P e58d56335e084712b0f4c8aecff74e8c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e58d56335e084712b0f4c8aecff74e8c, State: Initialized, Role: FOLLOWER
I20260925 16:35:33.760942 14795 consensus_queue.cc:260] T f4467bd68d5d461880f7a500d8215c6c P e58d56335e084712b0f4c8aecff74e8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb773df56da488da71618c4eb87301a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34417 } } peers { permanent_uuid: "e58d56335e084712b0f4c8aecff74e8c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38835 } } peers { permanent_uuid: "2dafcd8fb7e74e919d7ec6ae0665d657" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41139 } }
I20260925 16:35:33.761924 14795 ts_tablet_manager.cc:1428] T f4467bd68d5d461880f7a500d8215c6c P e58d56335e084712b0f4c8aecff74e8c: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260925 16:35:33.762246 14715 ts_manager.cc:194] Registered new tserver with Master: e58d56335e084712b0f4c8aecff74e8c (127.0.98.1:38835)
I20260925 16:35:33.766352 14715 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45253
I20260925 16:35:33.769186 14903 heartbeater.cc:503] Master 127.0.98.62:35813 was elected leader, sending a full tablet report...
Sep 25 16:35:33 dist-test-slave-65pf krb5kdc[14135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354133, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 16:35:33.776799 14910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:33.765698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38835 (local address 127.0.0.1:42466)
0925 16:35:33.766140 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:33.766193 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 16:35:33.766339 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:35:33.766969 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:35:33.766976 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:35:33.767469 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:35:33.767634 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:33.767643 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:33.768890 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:33.768898 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:33.769749 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:33.769760 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:33.770565 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:33.770617 (+ 52us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:33.770746 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:33.771467 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:35:33.771480 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:35:33.772948 (+ 1468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:35:33.775119 (+ 2171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:35:33.775127 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:35:33.775131 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:35:33.775364 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:35:33.775660 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:35:33.775662 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:35:33.775664 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:35:33.775748 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:35:33.776259 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:35:33.776264 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:35:33.776380 (+ 116us) client_negotiation.cc:769] Sending connection context
0925 16:35:33.776506 (+ 126us) client_negotiation.cc:240] Negotiation successful
0925 16:35:33.776660 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":126,"threads_started":1}
I20260925 16:35:33.776916 14911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:33.765811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42466 (local address 127.0.98.1:38835)
0925 16:35:33.766479 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0925 16:35:33.766491 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 16:35:33.766514 (+ 23us) server_negotiation.cc:407] Connection header received
0925 16:35:33.766610 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:35:33.766616 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:35:33.766804 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:35:33.766935 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:35:33.767844 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:33.768741 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:33.769893 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:33.770437 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:33.770545 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:33.773094 (+ 2549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:35:33.773103 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:35:33.773114 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:35:33.773139 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:35:33.774996 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:33.775474 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:33.775479 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:33.775481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:33.775571 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:33.775830 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:33.775833 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:33.775835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:33.776127 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:35:33.776250 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 16:35:33.776602 (+ 352us) server_negotiation.cc:299] Negotiation successful
0925 16:35:33.776763 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":91,"threads_started":1}
I20260925 16:35:33.777935 14837 tablet_service.cc:1514] Processing DeleteTablet for tablet f4467bd68d5d461880f7a500d8215c6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:35:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42466
I20260925 16:35:33.778944 14912 tablet_replica.cc:330] stopping tablet replica
I20260925 16:35:33.779275 14912 raft_consensus.cc:2229] T f4467bd68d5d461880f7a500d8215c6c P e58d56335e084712b0f4c8aecff74e8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:35:33.779533 14912 raft_consensus.cc:2258] T f4467bd68d5d461880f7a500d8215c6c P e58d56335e084712b0f4c8aecff74e8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:35:33.780709 14912 ts_tablet_manager.cc:1905] T f4467bd68d5d461880f7a500d8215c6c P e58d56335e084712b0f4c8aecff74e8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:35:33.784997 14912 ts_tablet_manager.cc:1918] T f4467bd68d5d461880f7a500d8215c6c P e58d56335e084712b0f4c8aecff74e8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:35:33.785094 14912 log.cc:1196] T f4467bd68d5d461880f7a500d8215c6c P e58d56335e084712b0f4c8aecff74e8c: Deleting WAL directory at /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-0/wal/wals/f4467bd68d5d461880f7a500d8215c6c
I20260925 16:35:33.785349 14912 ts_tablet_manager.cc:1939] T f4467bd68d5d461880f7a500d8215c6c P e58d56335e084712b0f4c8aecff74e8c: Deleting consensus metadata
I20260925 16:35:33.787364 14701 catalog_manager.cc:4841] TS e58d56335e084712b0f4c8aecff74e8c (127.0.98.1:38835): tablet f4467bd68d5d461880f7a500d8215c6c (table test-table [id=3298dde6786e4b84bc8c127ea989fb38]) successfully deleted
W20260925 16:35:33.867772 14908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:35:33.868052 14908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:35:33.868142 14908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:35:33.872210 14908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:35:33.872288 14908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:35:33.872373 14908 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 16:35:33.877696 14908 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:41139
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=33313
--tserver_master_addrs=127.0.98.62:35813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14908
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:35:33.878865 14908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:35:33.879951 14908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:35:33.889999 14918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:35:33.890067 14908 server_base.cc:1034] running on GCE node
W20260925 16:35:33.890101 14917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:35:33.890002 14920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:35:33.890688 14908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:35:33.891278 14908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:35:33.892439 14908 hybrid_clock.cc:648] HybridClock initialized: now 1790354133892425 us; error 28 us; skew 500 ppm
Sep 25 16:35:33 dist-test-slave-65pf krb5kdc[14135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354133, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:35:33.895206 14908 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 16:35:33.896145 14908 webserver.cc:460] Webserver started at http://127.0.98.2:33313/ using document root <none> and password file <none>
I20260925 16:35:33.896708 14908 fs_manager.cc:359] Metadata directory not provided
I20260925 16:35:33.896780 14908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:35:33.900542 14908 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260925 16:35:33.902522 14927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:35:33.903762 14908 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:35:33.903892 14908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-1/wal
uuid: "2dafcd8fb7e74e919d7ec6ae0665d657"
format_stamp: "Formatted at 2026-09-25 16:35:32 on dist-test-slave-65pf"
server_key: "2c7d016aadaddec123ce70e4b2071524"
server_key_iv: "17f97ab56abe3ca691678fef7ce2bfc9"
server_key_version: "encryptionkey@0"
I20260925 16:35:33.904417 14908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:35:33.930008 14908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:35:33.933113 14908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:35:33.933350 14908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:35:33.933957 14908 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:35:33.934857 14908 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:35:33.934927 14908 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:35:33.934994 14908 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:35:33.935047 14908 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:35:33.944851 14908 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41139
I20260925 16:35:33.944891 15040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41139 every 8 connection(s)
I20260925 16:35:33.945912 14908 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
I20260925 16:35:33.953979 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 14908
I20260925 16:35:33.954501 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:34417
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:35813
--webserver_port=44633
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 16:35:33 dist-test-slave-65pf krb5kdc[14135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354133, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:35:33.960450 14905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:33.948014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38947 (local address 127.0.98.62:35813)
0925 16:35:33.948187 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 16:35:33.948193 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:35:33.949176 (+ 983us) server_negotiation.cc:407] Connection header received
0925 16:35:33.950400 (+ 1224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:35:33.950404 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:35:33.950456 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:35:33.950533 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:35:33.951711 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:33.952267 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:33.953357 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:33.953607 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:33.953750 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:33.956551 (+ 2801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:35:33.956554 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:35:33.956556 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:35:33.956581 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:35:33.958254 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:33.958887 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:33.958891 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:33.958892 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:33.958966 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:33.959318 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:33.959321 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:33.959323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:33.959512 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:35:33.959639 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 16:35:33.960148 (+ 509us) server_negotiation.cc:299] Negotiation successful
0925 16:35:33.960282 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 16:35:33.961347 15043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:33.948531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35813 (local address 127.0.98.2:38947)
0925 16:35:33.949008 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:33.949052 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 16:35:33.950180 (+ 1128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:35:33.950725 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:35:33.950732 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:35:33.951078 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:35:33.951556 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:33.951570 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:33.952379 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:33.952382 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:33.953239 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:33.953247 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:33.953810 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:33.953814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:33.953973 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:33.954667 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:35:33.954688 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:35:33.956373 (+ 1685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:35:33.958380 (+ 2007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:35:33.958386 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:35:33.958402 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:35:33.958745 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:35:33.959065 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:35:33.959069 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:35:33.959071 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:35:33.959214 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:35:33.959618 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:35:33.959625 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:35:33.959909 (+ 284us) client_negotiation.cc:769] Sending connection context
0925 16:35:33.960098 (+ 189us) client_negotiation.cc:240] Negotiation successful
0925 16:35:33.960442 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":134,"threads_started":1}
I20260925 16:35:33.962729 15041 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35813
I20260925 16:35:33.962997 15041 heartbeater.cc:464] Registering TS with master...
I20260925 16:35:33.963522 15041 heartbeater.cc:511] Master 127.0.98.62:35813 requested a full tablet report, sending...
I20260925 16:35:33.964773 14715 ts_manager.cc:194] Registered new tserver with Master: 2dafcd8fb7e74e919d7ec6ae0665d657 (127.0.98.2:41139)
I20260925 16:35:33.965554 14715 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38947
W20260925 16:35:34.065447 15044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:35:34.065701 15044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:35:34.065742 15044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:35:34.069661 15044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:35:34.069715 15044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:35:34.069770 15044 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 16:35:34.074867 15044 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:34417
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44633
--tserver_master_addrs=127.0.98.62:35813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:35:34.075780 15044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:35:34.076699 15044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:35:34.086154 15052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:35:34.086191 15044 server_base.cc:1034] running on GCE node
W20260925 16:35:34.086310 15049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:35:34.086148 15050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:35:34.086853 15044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:35:34.087496 15044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:35:34.088692 15044 hybrid_clock.cc:648] HybridClock initialized: now 1790354134088659 us; error 45 us; skew 500 ppm
Sep 25 16:35:34 dist-test-slave-65pf krb5kdc[14135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354134, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:35:34.091513 15044 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 16:35:34.092290 15044 webserver.cc:460] Webserver started at http://127.0.98.3:44633/ using document root <none> and password file <none>
I20260925 16:35:34.092787 15044 fs_manager.cc:359] Metadata directory not provided
I20260925 16:35:34.092861 15044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:35:34.096508 15044 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:35:34.098431 15059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:35:34.099623 15044 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:35:34.099711 15044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-2/wal
uuid: "efb773df56da488da71618c4eb87301a"
format_stamp: "Formatted at 2026-09-25 16:35:32 on dist-test-slave-65pf"
server_key: "a2c7077d9c105203fbce111449069d7c"
server_key_iv: "a06bf4692104a4972717f8fc2c415386"
server_key_version: "encryptionkey@0"
I20260925 16:35:34.100221 15044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:35:34.111704 15044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:35:34.114332 15044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:35:34.114475 15044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:35:34.115033 15044 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:35:34.115761 15044 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:35:34.115813 15044 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:35:34.115859 15044 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:35:34.115881 15044 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:35:34.124348 15044 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34417
I20260925 16:35:34.124377 15172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34417 every 8 connection(s)
I20260925 16:35:34.125291 15044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
I20260925 16:35:34.131098 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 15044
Sep 25 16:35:34 dist-test-slave-65pf krb5kdc[14135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354134, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:35:34.139338 14905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:34.127078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48859 (local address 127.0.98.62:35813)
0925 16:35:34.127218 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 16:35:34.127222 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:35:34.127950 (+ 728us) server_negotiation.cc:407] Connection header received
0925 16:35:34.129126 (+ 1176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:35:34.129129 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:35:34.129178 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:35:34.129253 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:35:34.130362 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:34.130997 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:34.132421 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:34.132792 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:34.132929 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:34.135546 (+ 2617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:35:34.135549 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:35:34.135550 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:35:34.135571 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:35:34.137163 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:34.137802 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:34.137805 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:34.137807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:34.137886 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:34.138217 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:34.138222 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:34.138224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:34.138389 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:35:34.138492 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 16:35:34.139047 (+ 555us) server_negotiation.cc:299] Negotiation successful
0925 16:35:34.139168 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 16:35:34.139667 15176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:34.131786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58940 (local address 127.0.98.62:35813)
0925 16:35:34.132033 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0925 16:35:34.132037 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:35:34.132176 (+ 139us) server_negotiation.cc:407] Connection header received
0925 16:35:34.132257 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:35:34.132259 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:35:34.132306 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:35:34.132399 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:35:34.133236 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:34.133957 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:34.135110 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:34.135335 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:34.135441 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:34.136671 (+ 1230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:35:34.136675 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:35:34.136677 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:35:34.136715 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:35:34.138084 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:34.138489 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:34.138494 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:34.138496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:34.138572 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:34.138880 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:34.138883 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:34.138885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:34.139042 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:35:34.139253 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0925 16:35:34.139373 (+ 120us) server_negotiation.cc:299] Negotiation successful
0925 16:35:34.139491 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":71,"threads_started":1}
I20260925 16:35:34.140259 15175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:34.127384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35813 (local address 127.0.98.3:48859)
0925 16:35:34.127793 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:34.127832 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 16:35:34.128917 (+ 1085us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:35:34.129415 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:35:34.129421 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:35:34.129771 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:35:34.130219 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:34.130232 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:34.131118 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:34.131120 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:34.132283 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:34.132293 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:34.132945 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:34.132948 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:34.133083 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:34.133662 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:35:34.133680 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:35:34.135374 (+ 1694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:35:34.137282 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:35:34.137288 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:35:34.137302 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:35:34.137684 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:35:34.137981 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:35:34.137984 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:35:34.137986 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:35:34.138122 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:35:34.138497 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:35:34.138506 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:35:34.138812 (+ 306us) client_negotiation.cc:769] Sending connection context
0925 16:35:34.139000 (+ 188us) client_negotiation.cc:240] Negotiation successful
0925 16:35:34.139308 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":99,"threads_started":1}
I20260925 16:35:34.141315 15173 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35813
I20260925 16:35:34.141557 15173 heartbeater.cc:464] Registering TS with master...
I20260925 16:35:34.142025 15173 heartbeater.cc:511] Master 127.0.98.62:35813 requested a full tablet report, sending...
I20260925 16:35:34.143016 14714 ts_manager.cc:194] Registered new tserver with Master: efb773df56da488da71618c4eb87301a (127.0.98.3:34417)
I20260925 16:35:34.143580 14714 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48859
I20260925 16:35:34.153321 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 16:35:34.157944 15176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:34.153988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58942 (local address 127.0.98.62:35813)
0925 16:35:34.154121 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 16:35:34.154126 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:35:34.154366 (+ 240us) server_negotiation.cc:407] Connection header received
0925 16:35:34.154546 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:35:34.154548 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:35:34.154703 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:35:34.154815 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:35:34.155077 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:34.155606 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:34.156487 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:34.156721 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:34.156833 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:34.157179 (+ 346us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:35:34.157571 (+ 392us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:35:34.157709 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 16:35:34.157746 (+ 37us) server_negotiation.cc:299] Negotiation successful
0925 16:35:34.157804 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 16:35:34.158558 14714 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:58942:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 16:35:34.161859 14714 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:35:34.169842 14837 tablet_service.cc:1467] Processing CreateTablet for tablet 9e172f4b22cf445589557e76ab16a608 (DEFAULT_TABLE table=test-table [id=f2eca20f8736454888b6a5859d831ba6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:35:34.170554 14837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e172f4b22cf445589557e76ab16a608. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:35:34.173573 14795 tablet_bootstrap.cc:492] T 9e172f4b22cf445589557e76ab16a608 P e58d56335e084712b0f4c8aecff74e8c: Bootstrap starting.
I20260925 16:35:34.173619 14910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:34.168740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41139 (local address 127.0.0.1:58670)
0925 16:35:34.168868 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:34.168884 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:35:34.168964 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:35:34.169763 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:35:34.169766 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:35:34.169784 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:35:34.169895 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:34.169901 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:34.171306 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:34.171310 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:34.172150 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:34.172164 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:34.173176 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:34.173179 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:34.173271 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:34.173319 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 16:35:34.173406 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 16:35:34.173487 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 16:35:34.173696 15182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:34.168910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58670 (local address 127.0.98.2:41139)
0925 16:35:34.169249 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0925 16:35:34.169262 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 16:35:34.169290 (+ 28us) server_negotiation.cc:407] Connection header received
0925 16:35:34.169375 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:35:34.169381 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:35:34.169609 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:35:34.169785 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:35:34.169996 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:34.171176 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:34.172263 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:34.173073 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:34.173186 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:34.173340 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 16:35:34.173434 (+ 94us) server_negotiation.cc:299] Negotiation successful
0925 16:35:34.173543 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":127,"threads_started":1}
I20260925 16:35:34.174491 14795 tablet_bootstrap.cc:654] T 9e172f4b22cf445589557e76ab16a608 P e58d56335e084712b0f4c8aecff74e8c: Neither blocks nor log segments found. Creating new log.
I20260925 16:35:34.174489 15184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:34.169604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52620 (local address 127.0.98.3:34417)
0925 16:35:34.170181 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0925 16:35:34.170205 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0925 16:35:34.170225 (+ 20us) server_negotiation.cc:407] Connection header received
0925 16:35:34.170299 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:35:34.170303 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:35:34.170671 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:35:34.170846 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:35:34.170993 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:34.172021 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:34.173095 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:34.173912 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:34.174019 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:34.174147 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 16:35:34.174264 (+ 117us) server_negotiation.cc:299] Negotiation successful
0925 16:35:34.174354 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":163,"threads_started":1}
I20260925 16:35:34.174499 15183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:34.169188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34417 (local address 127.0.0.1:52620)
0925 16:35:34.169456 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:34.169470 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:35:34.169567 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:35:34.170791 (+ 1224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:35:34.170794 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:35:34.170830 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:35:34.170914 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:34.170919 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:34.172192 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:34.172198 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:34.172971 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:34.172979 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:34.174038 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:34.174042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:34.174138 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:34.174164 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 16:35:34.174242 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 16:35:34.174310 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":117,"threads_started":1}
I20260925 16:35:34.176117 14795 tablet_bootstrap.cc:492] T 9e172f4b22cf445589557e76ab16a608 P e58d56335e084712b0f4c8aecff74e8c: No bootstrap required, opened a new log
I20260925 16:35:34.176210 14795 ts_tablet_manager.cc:1397] T 9e172f4b22cf445589557e76ab16a608 P e58d56335e084712b0f4c8aecff74e8c: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260925 16:35:34.176645 14795 raft_consensus.cc:348] T 9e172f4b22cf445589557e76ab16a608 P e58d56335e084712b0f4c8aecff74e8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dafcd8fb7e74e919d7ec6ae0665d657" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41139 } } peers { permanent_uuid: "efb773df56da488da71618c4eb87301a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34417 } } peers { permanent_uuid: "e58d56335e084712b0f4c8aecff74e8c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38835 } }
I20260925 16:35:34.176766 14795 raft_consensus.cc:374] T 9e172f4b22cf445589557e76ab16a608 P e58d56335e084712b0f4c8aecff74e8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:35:34.176815 14795 raft_consensus.cc:729] T 9e172f4b22cf445589557e76ab16a608 P e58d56335e084712b0f4c8aecff74e8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e58d56335e084712b0f4c8aecff74e8c, State: Initialized, Role: FOLLOWER
I20260925 16:35:34.176941 14795 consensus_queue.cc:260] T 9e172f4b22cf445589557e76ab16a608 P e58d56335e084712b0f4c8aecff74e8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dafcd8fb7e74e919d7ec6ae0665d657" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41139 } } peers { permanent_uuid: "efb773df56da488da71618c4eb87301a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34417 } } peers { permanent_uuid: "e58d56335e084712b0f4c8aecff74e8c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38835 } }
I20260925 16:35:34.176651 14975 tablet_service.cc:1467] Processing CreateTablet for tablet 9e172f4b22cf445589557e76ab16a608 (DEFAULT_TABLE table=test-table [id=f2eca20f8736454888b6a5859d831ba6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:35:34.177199 14795 ts_tablet_manager.cc:1428] T 9e172f4b22cf445589557e76ab16a608 P e58d56335e084712b0f4c8aecff74e8c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 16:35:34.177699 14975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e172f4b22cf445589557e76ab16a608. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:35:34.177542 15107 tablet_service.cc:1467] Processing CreateTablet for tablet 9e172f4b22cf445589557e76ab16a608 (DEFAULT_TABLE table=test-table [id=f2eca20f8736454888b6a5859d831ba6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:35:34.178323 15107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e172f4b22cf445589557e76ab16a608. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:35:34.184942 15186 tablet_bootstrap.cc:492] T 9e172f4b22cf445589557e76ab16a608 P 2dafcd8fb7e74e919d7ec6ae0665d657: Bootstrap starting.
I20260925 16:35:34.184944 15187 tablet_bootstrap.cc:492] T 9e172f4b22cf445589557e76ab16a608 P efb773df56da488da71618c4eb87301a: Bootstrap starting.
I20260925 16:35:34.187450 15187 tablet_bootstrap.cc:654] T 9e172f4b22cf445589557e76ab16a608 P efb773df56da488da71618c4eb87301a: Neither blocks nor log segments found. Creating new log.
I20260925 16:35:34.187467 15186 tablet_bootstrap.cc:654] T 9e172f4b22cf445589557e76ab16a608 P 2dafcd8fb7e74e919d7ec6ae0665d657: Neither blocks nor log segments found. Creating new log.
I20260925 16:35:34.188172 15187 log.cc:824] T 9e172f4b22cf445589557e76ab16a608 P efb773df56da488da71618c4eb87301a: Log is configured to *not* fsync() on all Append() calls
I20260925 16:35:34.188172 15186 log.cc:824] T 9e172f4b22cf445589557e76ab16a608 P 2dafcd8fb7e74e919d7ec6ae0665d657: Log is configured to *not* fsync() on all Append() calls
I20260925 16:35:34.189973 15187 tablet_bootstrap.cc:492] T 9e172f4b22cf445589557e76ab16a608 P efb773df56da488da71618c4eb87301a: No bootstrap required, opened a new log
I20260925 16:35:34.190052 15186 tablet_bootstrap.cc:492] T 9e172f4b22cf445589557e76ab16a608 P 2dafcd8fb7e74e919d7ec6ae0665d657: No bootstrap required, opened a new log
I20260925 16:35:34.190174 15187 ts_tablet_manager.cc:1397] T 9e172f4b22cf445589557e76ab16a608 P efb773df56da488da71618c4eb87301a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 16:35:34.190254 15186 ts_tablet_manager.cc:1397] T 9e172f4b22cf445589557e76ab16a608 P 2dafcd8fb7e74e919d7ec6ae0665d657: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 16:35:34.193153 15187 raft_consensus.cc:348] T 9e172f4b22cf445589557e76ab16a608 P efb773df56da488da71618c4eb87301a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dafcd8fb7e74e919d7ec6ae0665d657" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41139 } } peers { permanent_uuid: "efb773df56da488da71618c4eb87301a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34417 } } peers { permanent_uuid: "e58d56335e084712b0f4c8aecff74e8c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38835 } }
I20260925 16:35:34.193153 15186 raft_consensus.cc:348] T 9e172f4b22cf445589557e76ab16a608 P 2dafcd8fb7e74e919d7ec6ae0665d657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dafcd8fb7e74e919d7ec6ae0665d657" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41139 } } peers { permanent_uuid: "efb773df56da488da71618c4eb87301a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34417 } } peers { permanent_uuid: "e58d56335e084712b0f4c8aecff74e8c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38835 } }
I20260925 16:35:34.193377 15187 raft_consensus.cc:374] T 9e172f4b22cf445589557e76ab16a608 P efb773df56da488da71618c4eb87301a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:35:34.193408 15186 raft_consensus.cc:374] T 9e172f4b22cf445589557e76ab16a608 P 2dafcd8fb7e74e919d7ec6ae0665d657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:35:34.193410 15187 raft_consensus.cc:729] T 9e172f4b22cf445589557e76ab16a608 P efb773df56da488da71618c4eb87301a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efb773df56da488da71618c4eb87301a, State: Initialized, Role: FOLLOWER
I20260925 16:35:34.193495 15186 raft_consensus.cc:729] T 9e172f4b22cf445589557e76ab16a608 P 2dafcd8fb7e74e919d7ec6ae0665d657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dafcd8fb7e74e919d7ec6ae0665d657, State: Initialized, Role: FOLLOWER
I20260925 16:35:34.193890 15187 consensus_queue.cc:260] T 9e172f4b22cf445589557e76ab16a608 P efb773df56da488da71618c4eb87301a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dafcd8fb7e74e919d7ec6ae0665d657" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41139 } } peers { permanent_uuid: "efb773df56da488da71618c4eb87301a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34417 } } peers { permanent_uuid: "e58d56335e084712b0f4c8aecff74e8c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38835 } }
I20260925 16:35:34.193898 15186 consensus_queue.cc:260] T 9e172f4b22cf445589557e76ab16a608 P 2dafcd8fb7e74e919d7ec6ae0665d657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dafcd8fb7e74e919d7ec6ae0665d657" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41139 } } peers { permanent_uuid: "efb773df56da488da71618c4eb87301a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34417 } } peers { permanent_uuid: "e58d56335e084712b0f4c8aecff74e8c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38835 } }
I20260925 16:35:34.194573 15041 heartbeater.cc:503] Master 127.0.98.62:35813 was elected leader, sending a full tablet report...
I20260925 16:35:34.194787 15173 heartbeater.cc:503] Master 127.0.98.62:35813 was elected leader, sending a full tablet report...
I20260925 16:35:34.194866 15187 ts_tablet_manager.cc:1428] T 9e172f4b22cf445589557e76ab16a608 P efb773df56da488da71618c4eb87301a: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 16:35:34.194867 15186 ts_tablet_manager.cc:1428] T 9e172f4b22cf445589557e76ab16a608 P 2dafcd8fb7e74e919d7ec6ae0665d657: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260925 16:35:34.196841 15042 tablet.cc:2369] T 9e172f4b22cf445589557e76ab16a608 P 2dafcd8fb7e74e919d7ec6ae0665d657: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 16:35:34.237872 14904 tablet.cc:2369] T 9e172f4b22cf445589557e76ab16a608 P e58d56335e084712b0f4c8aecff74e8c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:35:34.308312 14909 raft_consensus.cc:482] T 9e172f4b22cf445589557e76ab16a608 P e58d56335e084712b0f4c8aecff74e8c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:35:34.308555 14909 raft_consensus.cc:504] T 9e172f4b22cf445589557e76ab16a608 P e58d56335e084712b0f4c8aecff74e8c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dafcd8fb7e74e919d7ec6ae0665d657" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41139 } } peers { permanent_uuid: "efb773df56da488da71618c4eb87301a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34417 } } peers { permanent_uuid: "e58d56335e084712b0f4c8aecff74e8c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38835 } }
I20260925 16:35:34.309768 14909 leader_election.cc:288] T 9e172f4b22cf445589557e76ab16a608 P e58d56335e084712b0f4c8aecff74e8c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2dafcd8fb7e74e919d7ec6ae0665d657 (127.0.98.2:41139), efb773df56da488da71618c4eb87301a (127.0.98.3:34417)
I20260925 16:35:34.313764 15182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:34.310101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54057 (local address 127.0.98.2:41139)
0925 16:35:34.310229 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 16:35:34.310233 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:35:34.310319 (+ 86us) server_negotiation.cc:407] Connection header received
0925 16:35:34.310649 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:35:34.310652 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:35:34.310706 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:35:34.310803 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:35:34.311119 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:34.311595 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:34.312737 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:34.313139 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:34.313257 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:34.313335 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 16:35:34.313555 (+ 220us) server_negotiation.cc:299] Negotiation successful
0925 16:35:34.313639 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 16:35:34.313762 15184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:34.310257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57617 (local address 127.0.98.3:34417)
0925 16:35:34.310399 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 16:35:34.310404 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:35:34.310485 (+ 81us) server_negotiation.cc:407] Connection header received
0925 16:35:34.310639 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:35:34.310642 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:35:34.310685 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:35:34.310744 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:35:34.311076 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:34.311578 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:34.312729 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:34.313139 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:34.313257 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:34.313324 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 16:35:34.313551 (+ 227us) server_negotiation.cc:299] Negotiation successful
0925 16:35:34.313630 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 16:35:34.313800 15192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:34.309953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41139 (local address 127.0.98.1:54057)
0925 16:35:34.310206 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:34.310222 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:35:34.310426 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:35:34.310831 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:35:34.310835 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:35:34.310866 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:35:34.310974 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:34.310979 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:34.311723 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:34.311727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:34.312606 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:34.312614 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:34.313261 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:34.313265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:34.313399 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:34.313442 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 16:35:34.313536 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 16:35:34.313656 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":121,"thread_start_us":79,"threads_started":1}
I20260925 16:35:34.313839 15193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:34.310052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34417 (local address 127.0.98.1:57617)
0925 16:35:34.310405 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:34.310418 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:35:34.310500 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:35:34.310788 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:35:34.310793 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:35:34.310810 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:35:34.310960 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:34.310966 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:34.311700 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:34.311706 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:34.312605 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:34.312614 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:34.313261 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:34.313265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:34.313402 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:34.313444 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 16:35:34.313533 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 16:35:34.313658 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":98,"spinlock_wait_cycles":21376,"thread_start_us":72,"threads_started":1}
I20260925 16:35:34.314692 15127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e172f4b22cf445589557e76ab16a608" candidate_uuid: "e58d56335e084712b0f4c8aecff74e8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efb773df56da488da71618c4eb87301a" is_pre_election: true
I20260925 16:35:34.314884 14995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e172f4b22cf445589557e76ab16a608" candidate_uuid: "e58d56335e084712b0f4c8aecff74e8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dafcd8fb7e74e919d7ec6ae0665d657" is_pre_election: true
I20260925 16:35:34.315056 15127 raft_consensus.cc:2454] T 9e172f4b22cf445589557e76ab16a608 P efb773df56da488da71618c4eb87301a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e58d56335e084712b0f4c8aecff74e8c in term 0.
I20260925 16:35:34.315207 14995 raft_consensus.cc:2454] T 9e172f4b22cf445589557e76ab16a608 P 2dafcd8fb7e74e919d7ec6ae0665d657 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e58d56335e084712b0f4c8aecff74e8c in term 0.
I20260925 16:35:34.315735 14790 leader_election.cc:302] T 9e172f4b22cf445589557e76ab16a608 P e58d56335e084712b0f4c8aecff74e8c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e58d56335e084712b0f4c8aecff74e8c, efb773df56da488da71618c4eb87301a; no voters:
I20260925 16:35:34.316035 14909 raft_consensus.cc:2781] T 9e172f4b22cf445589557e76ab16a608 P e58d56335e084712b0f4c8aecff74e8c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:35:34.316126 14909 raft_consensus.cc:482] T 9e172f4b22cf445589557e76ab16a608 P e58d56335e084712b0f4c8aecff74e8c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:35:34.316171 14909 raft_consensus.cc:3037] T 9e172f4b22cf445589557e76ab16a608 P e58d56335e084712b0f4c8aecff74e8c [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:35:34.317163 14909 raft_consensus.cc:504] T 9e172f4b22cf445589557e76ab16a608 P e58d56335e084712b0f4c8aecff74e8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dafcd8fb7e74e919d7ec6ae0665d657" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41139 } } peers { permanent_uuid: "efb773df56da488da71618c4eb87301a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34417 } } peers { permanent_uuid: "e58d56335e084712b0f4c8aecff74e8c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38835 } }
I20260925 16:35:34.317495 14909 leader_election.cc:288] T 9e172f4b22cf445589557e76ab16a608 P e58d56335e084712b0f4c8aecff74e8c [CANDIDATE]: Term 1 election: Requested vote from peers 2dafcd8fb7e74e919d7ec6ae0665d657 (127.0.98.2:41139), efb773df56da488da71618c4eb87301a (127.0.98.3:34417)
I20260925 16:35:34.317843 14995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e172f4b22cf445589557e76ab16a608" candidate_uuid: "e58d56335e084712b0f4c8aecff74e8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dafcd8fb7e74e919d7ec6ae0665d657"
I20260925 16:35:34.317874 15127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e172f4b22cf445589557e76ab16a608" candidate_uuid: "e58d56335e084712b0f4c8aecff74e8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efb773df56da488da71618c4eb87301a"
I20260925 16:35:34.318003 14995 raft_consensus.cc:3037] T 9e172f4b22cf445589557e76ab16a608 P 2dafcd8fb7e74e919d7ec6ae0665d657 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:35:34.318017 15127 raft_consensus.cc:3037] T 9e172f4b22cf445589557e76ab16a608 P efb773df56da488da71618c4eb87301a [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:35:34.319096 14995 raft_consensus.cc:2454] T 9e172f4b22cf445589557e76ab16a608 P 2dafcd8fb7e74e919d7ec6ae0665d657 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e58d56335e084712b0f4c8aecff74e8c in term 1.
I20260925 16:35:34.319124 15127 raft_consensus.cc:2454] T 9e172f4b22cf445589557e76ab16a608 P efb773df56da488da71618c4eb87301a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e58d56335e084712b0f4c8aecff74e8c in term 1.
I20260925 16:35:34.319393 14789 leader_election.cc:302] T 9e172f4b22cf445589557e76ab16a608 P e58d56335e084712b0f4c8aecff74e8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dafcd8fb7e74e919d7ec6ae0665d657, e58d56335e084712b0f4c8aecff74e8c; no voters:
I20260925 16:35:34.319625 14909 raft_consensus.cc:2781] T 9e172f4b22cf445589557e76ab16a608 P e58d56335e084712b0f4c8aecff74e8c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:35:34.319828 14909 raft_consensus.cc:686] T 9e172f4b22cf445589557e76ab16a608 P e58d56335e084712b0f4c8aecff74e8c [term 1 LEADER]: Becoming Leader. State: Replica: e58d56335e084712b0f4c8aecff74e8c, State: Running, Role: LEADER
I20260925 16:35:34.320083 14909 consensus_queue.cc:237] T 9e172f4b22cf445589557e76ab16a608 P e58d56335e084712b0f4c8aecff74e8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dafcd8fb7e74e919d7ec6ae0665d657" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41139 } } peers { permanent_uuid: "efb773df56da488da71618c4eb87301a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34417 } } peers { permanent_uuid: "e58d56335e084712b0f4c8aecff74e8c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38835 } }
I20260925 16:35:34.323063 14715 catalog_manager.cc:5495] T 9e172f4b22cf445589557e76ab16a608 P e58d56335e084712b0f4c8aecff74e8c reported cstate change: term changed from 0 to 1, leader changed from <none> to e58d56335e084712b0f4c8aecff74e8c (127.0.98.1). New cstate: current_term: 1 leader_uuid: "e58d56335e084712b0f4c8aecff74e8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dafcd8fb7e74e919d7ec6ae0665d657" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efb773df56da488da71618c4eb87301a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e58d56335e084712b0f4c8aecff74e8c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38835 } health_report { overall_health: HEALTHY } } }
I20260925 16:35:34.337110 15197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:34.333586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42478 (local address 127.0.98.1:38835)
0925 16:35:34.333819 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0925 16:35:34.333828 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 16:35:34.333842 (+ 14us) server_negotiation.cc:407] Connection header received
0925 16:35:34.333924 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:35:34.333927 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:35:34.333975 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:35:34.334097 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:35:34.334339 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:34.334896 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:34.335699 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:34.335920 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:34.336029 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:34.336296 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:35:34.336693 (+ 397us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:35:34.336810 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 16:35:34.336884 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 16:35:34.336935 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":81,"threads_started":1}
I20260925 16:35:34.341084 14995 raft_consensus.cc:1261] T 9e172f4b22cf445589557e76ab16a608 P 2dafcd8fb7e74e919d7ec6ae0665d657 [term 1 FOLLOWER]: Refusing update from remote peer e58d56335e084712b0f4c8aecff74e8c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:35:34.341084 15127 raft_consensus.cc:1261] T 9e172f4b22cf445589557e76ab16a608 P efb773df56da488da71618c4eb87301a [term 1 FOLLOWER]: Refusing update from remote peer e58d56335e084712b0f4c8aecff74e8c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:35:34.341857 15194 consensus_queue.cc:1035] T 9e172f4b22cf445589557e76ab16a608 P e58d56335e084712b0f4c8aecff74e8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb773df56da488da71618c4eb87301a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:35:34.342057 14909 consensus_queue.cc:1035] T 9e172f4b22cf445589557e76ab16a608 P e58d56335e084712b0f4c8aecff74e8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dafcd8fb7e74e919d7ec6ae0665d657" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:35:34.351864 15199 mvcc.cc:205] Tried to move back new op lower bound from 7333290534253858816 to 7333290534177509376. Current Snapshot: MvccSnapshot[applied={T|T < 7333290534253858816}]
I20260925 16:35:34.352312 15198 mvcc.cc:205] Tried to move back new op lower bound from 7333290534253858816 to 7333290534177509376. Current Snapshot: MvccSnapshot[applied={T|T < 7333290534253858816 or (T in {7333290534253858816})}]
I20260925 16:35:34.352504 15200 mvcc.cc:205] Tried to move back new op lower bound from 7333290534253858816 to 7333290534177509376. Current Snapshot: MvccSnapshot[applied={T|T < 7333290534253858816}]
I20260925 16:35:34.359004 14714 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:58942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:35:34.359174 14714 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:58942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:35:34.362110 14714 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0b6a310d97744021b7dd800019cc5eef: Sending DeleteTablet for 3 replicas of tablet 9e172f4b22cf445589557e76ab16a608
I20260925 16:35:34.362881 14975 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e172f4b22cf445589557e76ab16a608 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:35:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58670
I20260925 16:35:34.363140 15107 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e172f4b22cf445589557e76ab16a608 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:35:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52620
I20260925 16:35:34.363210 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 14753
I20260925 16:35:34.363541 15208 tablet_replica.cc:330] T 9e172f4b22cf445589557e76ab16a608 P 2dafcd8fb7e74e919d7ec6ae0665d657: stopping tablet replica
I20260925 16:35:34.364080 15208 raft_consensus.cc:2229] T 9e172f4b22cf445589557e76ab16a608 P 2dafcd8fb7e74e919d7ec6ae0665d657 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:35:34.364125 15209 tablet_replica.cc:330] T 9e172f4b22cf445589557e76ab16a608 P efb773df56da488da71618c4eb87301a: stopping tablet replica
I20260925 16:35:34.364416 15209 raft_consensus.cc:2229] T 9e172f4b22cf445589557e76ab16a608 P efb773df56da488da71618c4eb87301a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:35:34.364421 15208 raft_consensus.cc:2258] T 9e172f4b22cf445589557e76ab16a608 P 2dafcd8fb7e74e919d7ec6ae0665d657 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:35:34.365062 15209 raft_consensus.cc:2258] T 9e172f4b22cf445589557e76ab16a608 P efb773df56da488da71618c4eb87301a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:35:34.365861 15208 ts_tablet_manager.cc:1905] T 9e172f4b22cf445589557e76ab16a608 P 2dafcd8fb7e74e919d7ec6ae0665d657: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:35:34.366298 15209 ts_tablet_manager.cc:1905] T 9e172f4b22cf445589557e76ab16a608 P efb773df56da488da71618c4eb87301a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:35:34.369342 15208 ts_tablet_manager.cc:1918] T 9e172f4b22cf445589557e76ab16a608 P 2dafcd8fb7e74e919d7ec6ae0665d657: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:35:34.369459 15208 log.cc:1196] T 9e172f4b22cf445589557e76ab16a608 P 2dafcd8fb7e74e919d7ec6ae0665d657: Deleting WAL directory at /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-1/wal/wals/9e172f4b22cf445589557e76ab16a608
I20260925 16:35:34.369805 15208 ts_tablet_manager.cc:1939] T 9e172f4b22cf445589557e76ab16a608 P 2dafcd8fb7e74e919d7ec6ae0665d657: Deleting consensus metadata
I20260925 16:35:34.370332 15209 ts_tablet_manager.cc:1918] T 9e172f4b22cf445589557e76ab16a608 P efb773df56da488da71618c4eb87301a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:35:34.370455 15209 log.cc:1196] T 9e172f4b22cf445589557e76ab16a608 P efb773df56da488da71618c4eb87301a: Deleting WAL directory at /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-2/wal/wals/9e172f4b22cf445589557e76ab16a608
I20260925 16:35:34.370802 15209 ts_tablet_manager.cc:1939] T 9e172f4b22cf445589557e76ab16a608 P efb773df56da488da71618c4eb87301a: Deleting consensus metadata
I20260925 16:35:34.371179 14699 catalog_manager.cc:4841] TS 2dafcd8fb7e74e919d7ec6ae0665d657 (127.0.98.2:41139): tablet 9e172f4b22cf445589557e76ab16a608 (table test-table [id=f2eca20f8736454888b6a5859d831ba6]) successfully deleted
I20260925 16:35:34.371948 14700 catalog_manager.cc:4841] TS efb773df56da488da71618c4eb87301a (127.0.98.3:34417): tablet 9e172f4b22cf445589557e76ab16a608 (table test-table [id=f2eca20f8736454888b6a5859d831ba6]) successfully deleted
I20260925 16:35:34.372490 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 14908
W20260925 16:35:34.372479 14701 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38835 (error 108)
I20260925 16:35:34.374096 15183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:34.373828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38835 (local address 127.0.0.1:42484)
0925 16:35:34.373958 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:34.374023 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 16:35:34.374259 14701 catalog_manager.cc:4584] TS e58d56335e084712b0f4c8aecff74e8c (127.0.98.1:38835): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9e172f4b22cf445589557e76ab16a608: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38835: connect: Connection refused (error 111)
I20260925 16:35:34.378806 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 15044
I20260925 16:35:34.384153 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 14682
I20260925 16:35:34.390828 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35813
--webserver_interface=127.0.98.62
--webserver_port=34251
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35813
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:35:34.499787 15211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:35:34.500051 15211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:35:34.500092 15211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:35:34.504037 15211 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:35:34.504093 15211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:35:34.504112 15211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:35:34.504129 15211 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:35:34.504144 15211 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:35:34.509452 15211 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=34251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:35:34.510448 15211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:35:34.511480 15211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:35:34.519749 15219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:35:34.519763 15217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:35:34.519799 15216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:35:34.519996 15211 server_base.cc:1034] running on GCE node
I20260925 16:35:34.520645 15211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:35:34.521723 15211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:35:34.522908 15211 hybrid_clock.cc:648] HybridClock initialized: now 1790354134522872 us; error 50 us; skew 500 ppm
Sep 25 16:35:34 dist-test-slave-65pf krb5kdc[14135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354134, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:35:34.525480 15211 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 16:35:34.526413 15211 webserver.cc:460] Webserver started at http://127.0.98.62:34251/ using document root <none> and password file <none>
I20260925 16:35:34.527035 15211 fs_manager.cc:359] Metadata directory not provided
I20260925 16:35:34.527098 15211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:35:34.531178 15211 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 16:35:34.533195 15226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:35:34.534420 15211 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:35:34.534543 15211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/master-0/wal
uuid: "0b6a310d97744021b7dd800019cc5eef"
format_stamp: "Formatted at 2026-09-25 16:35:32 on dist-test-slave-65pf"
server_key: "47e02e14eb355f1041a158b1faf73cc9"
server_key_iv: "1ea01d8ead1e8ff16019783038effe46"
server_key_version: "encryptionkey@0"
I20260925 16:35:34.535099 15211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:35:34.555279 15211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:35:34.558466 15211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:35:34.558723 15211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:35:34.565773 15211 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35813
I20260925 16:35:34.566051 15278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35813 every 8 connection(s)
I20260925 16:35:34.566938 15211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:35:34.571087 15279 sys_catalog.cc:263] Verifying existing consensus state
I20260925 16:35:34.572676 15279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b6a310d97744021b7dd800019cc5eef: Bootstrap starting.
I20260925 16:35:34.576994 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 15211
I20260925 16:35:34.577415 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:38835
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:35813
--webserver_port=33983
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 16:35:34.589787 15279 log.cc:824] T 00000000000000000000000000000000 P 0b6a310d97744021b7dd800019cc5eef: Log is configured to *not* fsync() on all Append() calls
I20260925 16:35:34.600802 15279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b6a310d97744021b7dd800019cc5eef: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 16:35:34.601217 15279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b6a310d97744021b7dd800019cc5eef: Bootstrap complete.
I20260925 16:35:34.605170 15279 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b6a310d97744021b7dd800019cc5eef [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b6a310d97744021b7dd800019cc5eef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35813 } }
I20260925 16:35:34.605578 15279 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b6a310d97744021b7dd800019cc5eef [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6a310d97744021b7dd800019cc5eef, State: Initialized, Role: FOLLOWER
I20260925 16:35:34.606112 15279 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b6a310d97744021b7dd800019cc5eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b6a310d97744021b7dd800019cc5eef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35813 } }
I20260925 16:35:34.606256 15279 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b6a310d97744021b7dd800019cc5eef [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:35:34.606345 15279 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b6a310d97744021b7dd800019cc5eef [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:35:34.606459 15279 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b6a310d97744021b7dd800019cc5eef [term 2 FOLLOWER]: Advancing to term 3
I20260925 16:35:34.608348 15279 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b6a310d97744021b7dd800019cc5eef [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b6a310d97744021b7dd800019cc5eef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35813 } }
I20260925 16:35:34.608696 15279 leader_election.cc:302] T 00000000000000000000000000000000 P 0b6a310d97744021b7dd800019cc5eef [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b6a310d97744021b7dd800019cc5eef; no voters:
I20260925 16:35:34.609025 15279 leader_election.cc:288] T 00000000000000000000000000000000 P 0b6a310d97744021b7dd800019cc5eef [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 16:35:34.609105 15283 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b6a310d97744021b7dd800019cc5eef [term 3 FOLLOWER]: Leader election won for term 3
I20260925 16:35:34.609308 15283 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b6a310d97744021b7dd800019cc5eef [term 3 LEADER]: Becoming Leader. State: Replica: 0b6a310d97744021b7dd800019cc5eef, State: Running, Role: LEADER
I20260925 16:35:34.609587 15283 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b6a310d97744021b7dd800019cc5eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b6a310d97744021b7dd800019cc5eef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35813 } }
I20260925 16:35:34.610234 15279 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b6a310d97744021b7dd800019cc5eef [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:35:34.610476 15283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b6a310d97744021b7dd800019cc5eef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b6a310d97744021b7dd800019cc5eef. Latest consensus state: current_term: 3 leader_uuid: "0b6a310d97744021b7dd800019cc5eef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b6a310d97744021b7dd800019cc5eef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35813 } } }
I20260925 16:35:34.610617 15283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b6a310d97744021b7dd800019cc5eef [sys.catalog]: This master's current role is: LEADER
I20260925 16:35:34.611049 15284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b6a310d97744021b7dd800019cc5eef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "0b6a310d97744021b7dd800019cc5eef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b6a310d97744021b7dd800019cc5eef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35813 } } }
I20260925 16:35:34.611208 15284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b6a310d97744021b7dd800019cc5eef [sys.catalog]: This master's current role is: LEADER
I20260925 16:35:34.612311 15287 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:35:34.617360 15287 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:35:34.617993 15287 catalog_manager.cc:1259] Loaded cluster ID: 6adea33195954dbeb67088ce5172c8ed
I20260925 16:35:34.618050 15287 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260925 16:35:34.619016 15287 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f52dcb829ad google::LogMessage::Fail() at ??:0
@ 0x7f52dcb8588a google::LogMessage::SendToLog() at ??:0
@ 0x7f52dcb824de google::LogMessage::Flush() at ??:0
@ 0x7f52dcb83a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f52e74c1fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f52e74c2d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f52e74bf579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f52e7503640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f52dedd43d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f52dedd4c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f52dedd63f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f52dedc5512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f52e04096db start_thread at ??:0
@ 0x7f52db68061f clone at ??:0
W20260925 16:35:34.707834 15281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:35:34.708105 15281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:35:34.708196 15281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:35:34.712190 15281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:35:34.712265 15281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:35:34.712360 15281 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 16:35:34.717583 15281 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:38835
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=33983
--tserver_master_addrs=127.0.98.62:35813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15281
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:35:34.718724 15281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:35:34.719735 15281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:35:34.729321 15321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:35:34.729342 15320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:35:34.729593 15323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:35:34.729653 15281 server_base.cc:1034] running on GCE node
I20260925 16:35:34.730088 15281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:35:34.730861 15281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:35:34.732059 15281 hybrid_clock.cc:648] HybridClock initialized: now 1790354134732022 us; error 49 us; skew 500 ppm
Sep 25 16:35:34 dist-test-slave-65pf krb5kdc[14135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354134, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:35:34.735226 15281 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 16:35:34.736321 15281 webserver.cc:460] Webserver started at http://127.0.98.1:33983/ using document root <none> and password file <none>
I20260925 16:35:34.737046 15281 fs_manager.cc:359] Metadata directory not provided
I20260925 16:35:34.737138 15281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:35:34.742149 15281 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260925 16:35:34.744760 15330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:35:34.746273 15281 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 16:35:34.746397 15281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-0/wal
uuid: "e58d56335e084712b0f4c8aecff74e8c"
format_stamp: "Formatted at 2026-09-25 16:35:32 on dist-test-slave-65pf"
server_key: "e8d8a16ebfeb835b45e1ac4d80f7e1a3"
server_key_iv: "4ac91e6f2b8d6148b67d33911daf5d94"
server_key_version: "encryptionkey@0"
I20260925 16:35:34.747067 15281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:35:34.765205 15281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:35:34.769008 15281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:35:34.769215 15281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:35:34.769954 15281 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:35:34.771353 15337 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 16:35:34.775516 15281 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 16:35:34.775590 15281 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260925 16:35:34.775657 15281 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 16:35:34.777954 15281 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 16:35:34.778023 15281 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260925 16:35:34.778316 15337 tablet_bootstrap.cc:492] T 9e172f4b22cf445589557e76ab16a608 P e58d56335e084712b0f4c8aecff74e8c: Bootstrap starting.
I20260925 16:35:34.788862 15281 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38835
I20260925 16:35:34.789252 15444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38835 every 8 connection(s)
I20260925 16:35:34.790043 15281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
I20260925 16:35:34.794028 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 15281
I20260925 16:35:34.794495 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:41139
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:35813
--webserver_port=33313
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 16:35:34.798171 15337 log.cc:824] T 9e172f4b22cf445589557e76ab16a608 P e58d56335e084712b0f4c8aecff74e8c: Log is configured to *not* fsync() on all Append() calls
I20260925 16:35:34.808226 15337 tablet_bootstrap.cc:492] T 9e172f4b22cf445589557e76ab16a608 P e58d56335e084712b0f4c8aecff74e8c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 16:35:34.808661 15337 tablet_bootstrap.cc:492] T 9e172f4b22cf445589557e76ab16a608 P e58d56335e084712b0f4c8aecff74e8c: Bootstrap complete.
I20260925 16:35:34.810075 15337 ts_tablet_manager.cc:1397] T 9e172f4b22cf445589557e76ab16a608 P e58d56335e084712b0f4c8aecff74e8c: Time spent bootstrapping tablet: real 0.032s user 0.020s sys 0.008s
I20260925 16:35:34.813256 15337 raft_consensus.cc:348] T 9e172f4b22cf445589557e76ab16a608 P e58d56335e084712b0f4c8aecff74e8c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dafcd8fb7e74e919d7ec6ae0665d657" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41139 } } peers { permanent_uuid: "efb773df56da488da71618c4eb87301a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34417 } } peers { permanent_uuid: "e58d56335e084712b0f4c8aecff74e8c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38835 } }
I20260925 16:35:34.813833 15337 raft_consensus.cc:729] T 9e172f4b22cf445589557e76ab16a608 P e58d56335e084712b0f4c8aecff74e8c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e58d56335e084712b0f4c8aecff74e8c, State: Initialized, Role: FOLLOWER
I20260925 16:35:34.814357 15337 consensus_queue.cc:260] T 9e172f4b22cf445589557e76ab16a608 P e58d56335e084712b0f4c8aecff74e8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dafcd8fb7e74e919d7ec6ae0665d657" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41139 } } peers { permanent_uuid: "efb773df56da488da71618c4eb87301a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34417 } } peers { permanent_uuid: "e58d56335e084712b0f4c8aecff74e8c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38835 } }
I20260925 16:35:34.815469 15337 ts_tablet_manager.cc:1428] T 9e172f4b22cf445589557e76ab16a608 P e58d56335e084712b0f4c8aecff74e8c: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
W20260925 16:35:34.856110 15447 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:35:34.792334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35813 (local address 127.0.98.1:59173)
0925 16:35:34.792962 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:34.793041 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0925 16:35:34.794259 (+ 1218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:35:34.855236 (+ 60977us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35813: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":158,"threads_started":1}
W20260925 16:35:34.856702 15445 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35813 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:35813: Client connection negotiation failed: client connection to 127.0.98.62:35813: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 16:35:34.857414 15447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:34.857164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35813 (local address 127.0.98.1:43403)
0925 16:35:34.857293 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:34.857364 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260925 16:35:34.858115 15447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:34.857923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35813 (local address 127.0.98.1:56611)
0925 16:35:34.858023 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:34.858074 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 16:35:34.858299 15445 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 16:35:34.914228 15448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:35:34.914539 15448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:35:34.914645 15448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:35:34.918650 15448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:35:34.918745 15448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:35:34.918835 15448 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 16:35:34.923996 15448 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:41139
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=33313
--tserver_master_addrs=127.0.98.62:35813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15448
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:35:34.925072 15448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:35:34.926266 15448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:35:34.935873 15456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:35:34.935946 15455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:35:34.936033 15448 server_base.cc:1034] running on GCE node
W20260925 16:35:34.935881 15458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:35:34.936609 15448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:35:34.937218 15448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:35:34.938398 15448 hybrid_clock.cc:648] HybridClock initialized: now 1790354134938383 us; error 30 us; skew 500 ppm
Sep 25 16:35:34 dist-test-slave-65pf krb5kdc[14135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354134, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:35:34.941174 15448 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 16:35:34.942061 15448 webserver.cc:460] Webserver started at http://127.0.98.2:33313/ using document root <none> and password file <none>
I20260925 16:35:34.942679 15448 fs_manager.cc:359] Metadata directory not provided
I20260925 16:35:34.942744 15448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:35:34.946573 15448 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 16:35:34.948645 15465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:35:34.949869 15448 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:35:34.950003 15448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-1/wal
uuid: "2dafcd8fb7e74e919d7ec6ae0665d657"
format_stamp: "Formatted at 2026-09-25 16:35:32 on dist-test-slave-65pf"
server_key: "2c7d016aadaddec123ce70e4b2071524"
server_key_iv: "17f97ab56abe3ca691678fef7ce2bfc9"
server_key_version: "encryptionkey@0"
I20260925 16:35:34.950555 15448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:35:34.973989 15448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:35:34.977370 15448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:35:34.977586 15448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:35:34.978188 15448 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:35:34.979024 15448 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:35:34.979096 15448 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:35:34.979167 15448 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:35:34.979212 15448 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:35:34.987983 15448 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41139
I20260925 16:35:34.988009 15578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41139 every 8 connection(s)
I20260925 16:35:34.988960 15448 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
I20260925 16:35:34.992726 15581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:34.991120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35813 (local address 127.0.98.2:35439)
0925 16:35:34.991555 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:34.992047 (+ 492us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":106,"threads_started":1}
I20260925 16:35:34.993042 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 15448
W20260925 16:35:34.993248 15579 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35813 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:35813: Client connection negotiation failed: client connection to 127.0.98.62:35813: connect: Connection refused (error 111)
I20260925 16:35:34.993538 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:34417
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:35813
--webserver_port=44633
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 16:35:34.994017 15581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:34.993668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35813 (local address 127.0.98.2:38041)
0925 16:35:34.993860 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:34.993918 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260925 16:35:34.994726 15581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:34.994438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35813 (local address 127.0.98.2:45319)
0925 16:35:34.994567 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:34.994649 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 16:35:34.994921 15579 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 16:35:35.101284 15582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:35:35.101552 15582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:35:35.101594 15582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:35:35.105422 15582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:35:35.105471 15582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:35:35.105530 15582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 16:35:35.110662 15582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:34417
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44633
--tserver_master_addrs=127.0.98.62:35813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:35:35.111604 15582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:35:35.112519 15582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 16:35:35.121963 15582 server_base.cc:1034] running on GCE node
W20260925 16:35:35.121892 15587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:35:35.121846 15588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:35:35.121870 15590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:35:35.122646 15582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:35:35.123344 15582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:35:35.124531 15582 hybrid_clock.cc:648] HybridClock initialized: now 1790354135124494 us; error 49 us; skew 500 ppm
Sep 25 16:35:35 dist-test-slave-65pf krb5kdc[14135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354135, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:35:35.127205 15582 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 16:35:35.128049 15582 webserver.cc:460] Webserver started at http://127.0.98.3:44633/ using document root <none> and password file <none>
I20260925 16:35:35.128612 15582 fs_manager.cc:359] Metadata directory not provided
I20260925 16:35:35.128706 15582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:35:35.132563 15582 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260925 16:35:35.134485 15597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:35:35.135704 15582 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260925 16:35:35.135834 15582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-2/wal
uuid: "efb773df56da488da71618c4eb87301a"
format_stamp: "Formatted at 2026-09-25 16:35:32 on dist-test-slave-65pf"
server_key: "a2c7077d9c105203fbce111449069d7c"
server_key_iv: "a06bf4692104a4972717f8fc2c415386"
server_key_version: "encryptionkey@0"
I20260925 16:35:35.136360 15582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:35:35.149822 15582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:35:35.152674 15582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:35:35.152907 15582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:35:35.153539 15582 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:35:35.154330 15582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:35:35.154397 15582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:35:35.154466 15582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:35:35.154510 15582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:35:35.163466 15582 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34417
I20260925 16:35:35.163486 15710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34417 every 8 connection(s)
I20260925 16:35:35.164481 15582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
I20260925 16:35:35.168212 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:35.166541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35813 (local address 127.0.98.3:35509)
0925 16:35:35.167040 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:35.167528 (+ 488us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":96,"threads_started":1}
W20260925 16:35:35.168663 15711 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35813 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:35813: Client connection negotiation failed: client connection to 127.0.98.62:35813: connect: Connection refused (error 111)
I20260925 16:35:35.169246 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:35.169060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35813 (local address 127.0.98.3:37267)
0925 16:35:35.169157 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:35.169200 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 16:35:35.169771 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:35.169603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35813 (local address 127.0.98.3:36461)
0925 16:35:35.169691 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:35.169729 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 16:35:35.169919 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 15582
W20260925 16:35:35.169950 15711 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 16:35:35.859274 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:35.858874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35813 (local address 127.0.98.1:56251)
0925 16:35:35.859108 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:35.859162 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":69,"threads_started":1}
I20260925 16:35:35.995952 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:35.995535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35813 (local address 127.0.98.2:39063)
0925 16:35:35.995783 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:35.995847 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":70,"threads_started":1}
I20260925 16:35:36.170967 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:36.170513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35813 (local address 127.0.98.3:50317)
0925 16:35:36.170806 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:36.170870 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":114,"threads_started":1}
I20260925 16:35:36.231319 15718 raft_consensus.cc:482] T 9e172f4b22cf445589557e76ab16a608 P e58d56335e084712b0f4c8aecff74e8c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:35:36.231519 15718 raft_consensus.cc:504] T 9e172f4b22cf445589557e76ab16a608 P e58d56335e084712b0f4c8aecff74e8c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dafcd8fb7e74e919d7ec6ae0665d657" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41139 } } peers { permanent_uuid: "efb773df56da488da71618c4eb87301a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34417 } } peers { permanent_uuid: "e58d56335e084712b0f4c8aecff74e8c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38835 } }
I20260925 16:35:36.232632 15718 leader_election.cc:288] T 9e172f4b22cf445589557e76ab16a608 P e58d56335e084712b0f4c8aecff74e8c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2dafcd8fb7e74e919d7ec6ae0665d657 (127.0.98.2:41139), efb773df56da488da71618c4eb87301a (127.0.98.3:34417)
Sep 25 16:35:36 dist-test-slave-65pf krb5kdc[14135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354134, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 25 16:35:36 dist-test-slave-65pf krb5kdc[14135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354134, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260925 16:35:36.247561 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:36.232886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34417 (local address 127.0.98.1:53871)
0925 16:35:36.233247 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:36.233261 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:35:36.233353 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:35:36.235185 (+ 1832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:35:36.235188 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:35:36.235449 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:35:36.235921 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:36.235936 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:36.237467 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:36.237473 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:36.238422 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:36.238432 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:36.239200 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:36.239204 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:36.239371 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:36.240114 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:35:36.240126 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:35:36.241776 (+ 1650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:35:36.245218 (+ 3442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:35:36.245220 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:35:36.245222 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:35:36.245395 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:35:36.246125 (+ 730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:35:36.246127 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:35:36.246128 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:35:36.246197 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:35:36.246927 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:35:36.246931 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:35:36.247023 (+ 92us) client_negotiation.cc:769] Sending connection context
0925 16:35:36.247120 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 16:35:36.247339 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":156,"spinlock_wait_cycles":1792,"thread_start_us":79,"threads_started":1}
I20260925 16:35:36.247630 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:36.232965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44341 (local address 127.0.98.2:41139)
0925 16:35:36.233396 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0925 16:35:36.233428 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0925 16:35:36.233480 (+ 52us) server_negotiation.cc:407] Connection header received
0925 16:35:36.234157 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:35:36.234162 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:35:36.234499 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:35:36.234858 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:35:36.236115 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:36.237329 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:36.238572 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:36.239051 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:36.239232 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:36.242366 (+ 3134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:35:36.242374 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:35:36.242391 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:35:36.242426 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:35:36.244726 (+ 2300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:36.245236 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:36.245240 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:36.245244 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:36.245376 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:36.245795 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:36.245799 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:36.245801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:36.246383 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:35:36.246518 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 16:35:36.247113 (+ 595us) server_negotiation.cc:299] Negotiation successful
0925 16:35:36.247427 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":136,"threads_started":1}
I20260925 16:35:36.247682 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:36.233178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53871 (local address 127.0.98.3:34417)
0925 16:35:36.233442 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0925 16:35:36.233460 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 16:35:36.233493 (+ 33us) server_negotiation.cc:407] Connection header received
0925 16:35:36.234154 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:35:36.234159 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:35:36.234498 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:35:36.234811 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:35:36.236112 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:36.237322 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:36.238558 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:36.239050 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:36.239250 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:36.242371 (+ 3121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:35:36.242376 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:35:36.242391 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:35:36.242426 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:35:36.244726 (+ 2300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:36.245869 (+ 1143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:36.245875 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:36.245880 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:36.246010 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:36.246285 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:36.246290 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:36.246291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:36.246803 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:35:36.246905 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 16:35:36.247257 (+ 352us) server_negotiation.cc:299] Negotiation successful
0925 16:35:36.247494 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":77,"threads_started":1}
I20260925 16:35:36.248337 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:36.232837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41139 (local address 127.0.98.1:44341)
0925 16:35:36.232960 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:36.232980 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 16:35:36.233096 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:35:36.234924 (+ 1828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:35:36.234931 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:35:36.235452 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:35:36.235921 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:36.235936 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:36.237468 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:36.237473 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:36.238425 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:36.238435 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:36.239204 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:36.239208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:36.239374 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:36.240054 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:35:36.240071 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:35:36.242145 (+ 2074us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:35:36.244861 (+ 2716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:35:36.244866 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:35:36.244877 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:35:36.245129 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:35:36.245507 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:35:36.245510 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:35:36.245512 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:35:36.245678 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:35:36.246504 (+ 826us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:35:36.246509 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:35:36.246783 (+ 274us) client_negotiation.cc:769] Sending connection context
0925 16:35:36.246975 (+ 192us) client_negotiation.cc:240] Negotiation successful
0925 16:35:36.248185 (+ 1210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":75,"spinlock_wait_cycles":1280}
W20260925 16:35:36.250554 15332 leader_election.cc:341] T 9e172f4b22cf445589557e76ab16a608 P e58d56335e084712b0f4c8aecff74e8c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer efb773df56da488da71618c4eb87301a (127.0.98.3:34417): Not found: Tablet not found: 9e172f4b22cf445589557e76ab16a608
W20260925 16:35:36.250752 15331 leader_election.cc:341] T 9e172f4b22cf445589557e76ab16a608 P e58d56335e084712b0f4c8aecff74e8c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2dafcd8fb7e74e919d7ec6ae0665d657 (127.0.98.2:41139): Not found: Tablet not found: 9e172f4b22cf445589557e76ab16a608
I20260925 16:35:36.250898 15331 leader_election.cc:302] T 9e172f4b22cf445589557e76ab16a608 P e58d56335e084712b0f4c8aecff74e8c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e58d56335e084712b0f4c8aecff74e8c; no voters: 2dafcd8fb7e74e919d7ec6ae0665d657, efb773df56da488da71618c4eb87301a
I20260925 16:35:36.251183 15718 raft_consensus.cc:2726] T 9e172f4b22cf445589557e76ab16a608 P e58d56335e084712b0f4c8aecff74e8c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 16:35:36.860634 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:36.860213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35813 (local address 127.0.98.1:44237)
0925 16:35:36.860473 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:36.860538 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":72,"threads_started":1}
I20260925 16:35:36.997243 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:36.996780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35813 (local address 127.0.98.2:47395)
0925 16:35:36.997087 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:36.997151 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":86,"threads_started":1}
I20260925 16:35:37.172179 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:37.171784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35813 (local address 127.0.98.3:55377)
0925 16:35:37.172031 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:37.172093 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":72,"threads_started":1}
I20260925 16:35:37.806629 15725 raft_consensus.cc:482] T 9e172f4b22cf445589557e76ab16a608 P e58d56335e084712b0f4c8aecff74e8c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:35:37.806809 15725 raft_consensus.cc:504] T 9e172f4b22cf445589557e76ab16a608 P e58d56335e084712b0f4c8aecff74e8c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dafcd8fb7e74e919d7ec6ae0665d657" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41139 } } peers { permanent_uuid: "efb773df56da488da71618c4eb87301a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34417 } } peers { permanent_uuid: "e58d56335e084712b0f4c8aecff74e8c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38835 } }
I20260925 16:35:37.807262 15725 leader_election.cc:288] T 9e172f4b22cf445589557e76ab16a608 P e58d56335e084712b0f4c8aecff74e8c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2dafcd8fb7e74e919d7ec6ae0665d657 (127.0.98.2:41139), efb773df56da488da71618c4eb87301a (127.0.98.3:34417)
W20260925 16:35:37.808099 15331 leader_election.cc:341] T 9e172f4b22cf445589557e76ab16a608 P e58d56335e084712b0f4c8aecff74e8c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2dafcd8fb7e74e919d7ec6ae0665d657 (127.0.98.2:41139): Not found: Tablet not found: 9e172f4b22cf445589557e76ab16a608
W20260925 16:35:37.808192 15332 leader_election.cc:341] T 9e172f4b22cf445589557e76ab16a608 P e58d56335e084712b0f4c8aecff74e8c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer efb773df56da488da71618c4eb87301a (127.0.98.3:34417): Not found: Tablet not found: 9e172f4b22cf445589557e76ab16a608
I20260925 16:35:37.808308 15332 leader_election.cc:302] T 9e172f4b22cf445589557e76ab16a608 P e58d56335e084712b0f4c8aecff74e8c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e58d56335e084712b0f4c8aecff74e8c; no voters: 2dafcd8fb7e74e919d7ec6ae0665d657, efb773df56da488da71618c4eb87301a
I20260925 16:35:37.808493 15725 raft_consensus.cc:2726] T 9e172f4b22cf445589557e76ab16a608 P e58d56335e084712b0f4c8aecff74e8c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 16:35:37.861802 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:37.861397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35813 (local address 127.0.98.1:46695)
0925 16:35:37.861654 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:37.861709 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":100,"threads_started":1}
I20260925 16:35:37.998749 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:37.998187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35813 (local address 127.0.98.2:37239)
0925 16:35:37.998489 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:37.998554 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":92,"threads_started":1}
I20260925 16:35:38.173477 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:38.173026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35813 (local address 127.0.98.3:52049)
0925 16:35:38.173314 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:38.173379 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":113,"threads_started":1}
I20260925 16:35:38.863250 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:38.862806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35813 (local address 127.0.98.1:34281)
0925 16:35:38.863090 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:38.863153 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":78,"threads_started":1}
I20260925 16:35:39.000157 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:38.999700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35813 (local address 127.0.98.2:56981)
0925 16:35:38.999991 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:39.000057 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":92,"threads_started":1}
I20260925 16:35:39.174794 15731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:39.174350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35813 (local address 127.0.98.3:54507)
0925 16:35:39.174642 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:39.174702 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":73,"threads_started":1}
I20260925 16:35:39.864583 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:39.864135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35813 (local address 127.0.98.1:47375)
0925 16:35:39.864425 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:39.864484 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":73,"threads_started":1}
I20260925 16:35:40.001434 15733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:40.000978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35813 (local address 127.0.98.2:39751)
0925 16:35:40.001269 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:40.001330 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":87,"threads_started":1}
I20260925 16:35:40.175954 15734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:40.175551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35813 (local address 127.0.98.3:36899)
0925 16:35:40.175797 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:40.175851 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":72,"threads_started":1}
I20260925 16:35:40.354347 15735 raft_consensus.cc:482] T 9e172f4b22cf445589557e76ab16a608 P e58d56335e084712b0f4c8aecff74e8c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:35:40.354521 15735 raft_consensus.cc:504] T 9e172f4b22cf445589557e76ab16a608 P e58d56335e084712b0f4c8aecff74e8c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dafcd8fb7e74e919d7ec6ae0665d657" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41139 } } peers { permanent_uuid: "efb773df56da488da71618c4eb87301a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34417 } } peers { permanent_uuid: "e58d56335e084712b0f4c8aecff74e8c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38835 } }
I20260925 16:35:40.355078 15735 leader_election.cc:288] T 9e172f4b22cf445589557e76ab16a608 P e58d56335e084712b0f4c8aecff74e8c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2dafcd8fb7e74e919d7ec6ae0665d657 (127.0.98.2:41139), efb773df56da488da71618c4eb87301a (127.0.98.3:34417)
W20260925 16:35:40.355844 15331 leader_election.cc:341] T 9e172f4b22cf445589557e76ab16a608 P e58d56335e084712b0f4c8aecff74e8c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2dafcd8fb7e74e919d7ec6ae0665d657 (127.0.98.2:41139): Not found: Tablet not found: 9e172f4b22cf445589557e76ab16a608
W20260925 16:35:40.355964 15332 leader_election.cc:341] T 9e172f4b22cf445589557e76ab16a608 P e58d56335e084712b0f4c8aecff74e8c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer efb773df56da488da71618c4eb87301a (127.0.98.3:34417): Not found: Tablet not found: 9e172f4b22cf445589557e76ab16a608
I20260925 16:35:40.356015 15331 leader_election.cc:302] T 9e172f4b22cf445589557e76ab16a608 P e58d56335e084712b0f4c8aecff74e8c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e58d56335e084712b0f4c8aecff74e8c; no voters: 2dafcd8fb7e74e919d7ec6ae0665d657, efb773df56da488da71618c4eb87301a
I20260925 16:35:40.356189 15735 raft_consensus.cc:2726] T 9e172f4b22cf445589557e76ab16a608 P e58d56335e084712b0f4c8aecff74e8c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 16:35:40.865821 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:40.865403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35813 (local address 127.0.98.1:35249)
0925 16:35:40.865671 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:40.865728 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":72,"threads_started":1}
I20260925 16:35:41.002764 15737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:41.002241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35813 (local address 127.0.98.2:36763)
0925 16:35:41.002546 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:41.002669 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":74,"threads_started":1}
I20260925 16:35:41.177175 15738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:41.176759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35813 (local address 127.0.98.3:55045)
0925 16:35:41.177008 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:41.177064 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":94,"threads_started":1}
I20260925 16:35:41.867033 15739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:41.866624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35813 (local address 127.0.98.1:53697)
0925 16:35:41.866886 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:41.866939 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":74,"threads_started":1}
I20260925 16:35:42.004088 15740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:42.003594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35813 (local address 127.0.98.2:41373)
0925 16:35:42.003918 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:42.003982 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":70,"threads_started":1}
I20260925 16:35:42.178468 15741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:42.177975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35813 (local address 127.0.98.3:53967)
0925 16:35:42.178304 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:42.178365 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":74,"threads_started":1}
I20260925 16:35:42.868125 15742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:42.867721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35813 (local address 127.0.98.1:54061)
0925 16:35:42.867976 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:42.868030 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":78,"threads_started":1}
I20260925 16:35:43.005324 15743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:43.004847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35813 (local address 127.0.98.2:38861)
0925 16:35:43.005174 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:43.005233 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":71,"threads_started":1}
I20260925 16:35:43.179597 15744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:43.179211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35813 (local address 127.0.98.3:42781)
0925 16:35:43.179449 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:43.179505 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":70,"threads_started":1}
I20260925 16:35:43.869246 15745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:43.868857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35813 (local address 127.0.98.1:42029)
0925 16:35:43.869101 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:43.869153 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":87,"threads_started":1}
I20260925 16:35:44.006664 15746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:44.006143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35813 (local address 127.0.98.2:51631)
0925 16:35:44.006452 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:44.006503 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":103,"threads_started":1}
I20260925 16:35:44.180707 15747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:44.180322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35813 (local address 127.0.98.3:54901)
0925 16:35:44.180569 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:44.180617 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":92,"threads_started":1}
I20260925 16:35:44.870728 15748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:44.870249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35813 (local address 127.0.98.1:53783)
0925 16:35:44.870510 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:44.870624 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":97,"threads_started":1}
I20260925 16:35:45.007767 15749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:45.007368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35813 (local address 127.0.98.2:39281)
0925 16:35:45.007623 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:45.007678 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":80,"threads_started":1}
I20260925 16:35:45.181885 15750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:45.181482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35813 (local address 127.0.98.3:33819)
0925 16:35:45.181738 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:45.181792 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":85,"threads_started":1}
I20260925 16:35:45.311928 15751 raft_consensus.cc:482] T 9e172f4b22cf445589557e76ab16a608 P e58d56335e084712b0f4c8aecff74e8c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:35:45.312103 15751 raft_consensus.cc:504] T 9e172f4b22cf445589557e76ab16a608 P e58d56335e084712b0f4c8aecff74e8c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dafcd8fb7e74e919d7ec6ae0665d657" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41139 } } peers { permanent_uuid: "efb773df56da488da71618c4eb87301a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34417 } } peers { permanent_uuid: "e58d56335e084712b0f4c8aecff74e8c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38835 } }
I20260925 16:35:45.312573 15751 leader_election.cc:288] T 9e172f4b22cf445589557e76ab16a608 P e58d56335e084712b0f4c8aecff74e8c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2dafcd8fb7e74e919d7ec6ae0665d657 (127.0.98.2:41139), efb773df56da488da71618c4eb87301a (127.0.98.3:34417)
W20260925 16:35:45.313417 15331 leader_election.cc:341] T 9e172f4b22cf445589557e76ab16a608 P e58d56335e084712b0f4c8aecff74e8c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2dafcd8fb7e74e919d7ec6ae0665d657 (127.0.98.2:41139): Not found: Tablet not found: 9e172f4b22cf445589557e76ab16a608
W20260925 16:35:45.313505 15332 leader_election.cc:341] T 9e172f4b22cf445589557e76ab16a608 P e58d56335e084712b0f4c8aecff74e8c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer efb773df56da488da71618c4eb87301a (127.0.98.3:34417): Not found: Tablet not found: 9e172f4b22cf445589557e76ab16a608
I20260925 16:35:45.313611 15332 leader_election.cc:302] T 9e172f4b22cf445589557e76ab16a608 P e58d56335e084712b0f4c8aecff74e8c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e58d56335e084712b0f4c8aecff74e8c; no voters: 2dafcd8fb7e74e919d7ec6ae0665d657, efb773df56da488da71618c4eb87301a
I20260925 16:35:45.313756 15751 raft_consensus.cc:2726] T 9e172f4b22cf445589557e76ab16a608 P e58d56335e084712b0f4c8aecff74e8c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 16:35:45.871964 15752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:45.871515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35813 (local address 127.0.98.1:51681)
0925 16:35:45.871777 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:45.871852 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":86,"threads_started":1}
I20260925 16:35:46.008968 15753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:46.008530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35813 (local address 127.0.98.2:59917)
0925 16:35:46.008793 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:46.008855 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":74,"threads_started":1}
I20260925 16:35:46.183126 15754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:46.182641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35813 (local address 127.0.98.3:60263)
0925 16:35:46.182941 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:46.183024 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":85,"threads_started":1}
I20260925 16:35:46.873067 15755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:46.872674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35813 (local address 127.0.98.1:33195)
0925 16:35:46.872919 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:46.872972 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":72,"threads_started":1}
I20260925 16:35:47.010165 15756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:47.009744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35813 (local address 127.0.98.2:60925)
0925 16:35:47.010015 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:47.010074 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":94,"threads_started":1}
I20260925 16:35:47.184173 15757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:47.183799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35813 (local address 127.0.98.3:59699)
0925 16:35:47.184036 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:47.184086 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":85,"threads_started":1}
I20260925 16:35:47.874236 15758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:47.873858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35813 (local address 127.0.98.1:44839)
0925 16:35:47.874092 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:47.874144 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":75,"threads_started":1}
I20260925 16:35:48.011401 15759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:48.011010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35813 (local address 127.0.98.2:43495)
0925 16:35:48.011256 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:48.011309 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":85,"threads_started":1}
I20260925 16:35:48.185506 15760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:48.185026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35813 (local address 127.0.98.3:47793)
0925 16:35:48.185343 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:48.185409 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":98,"threads_started":1}
I20260925 16:35:48.875490 15761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:48.875075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35813 (local address 127.0.98.1:33681)
0925 16:35:48.875332 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:48.875392 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":75,"threads_started":1}
I20260925 16:35:49.012663 15762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:49.012188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35813 (local address 127.0.98.2:60293)
0925 16:35:49.012469 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:49.012530 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":93,"threads_started":1}
I20260925 16:35:49.186981 15763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:49.186378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35813 (local address 127.0.98.3:58353)
0925 16:35:49.186738 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:49.186799 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":79,"threads_started":1}
I20260925 16:35:49.876964 15764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:49.876548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35813 (local address 127.0.98.1:45873)
0925 16:35:49.876801 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:49.876875 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":70,"threads_started":1}
I20260925 16:35:50.014149 15765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:50.013645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35813 (local address 127.0.98.2:34165)
0925 16:35:50.013926 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:50.013996 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":111,"threads_started":1}
I20260925 16:35:50.176216 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 15281
I20260925 16:35:50.182693 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 15448
I20260925 16:35:50.188401 15766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:50.187963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35813 (local address 127.0.98.3:33029)
0925 16:35:50.188230 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:50.188293 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":99,"threads_started":1}
I20260925 16:35:50.188588 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 15582
2026-09-25T16:35:50Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19249 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:35:50 dist-test-slave-65pf krb5kdc[15769](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:35:50 dist-test-slave-65pf krb5kdc[15769](info): set up 2 sockets
Sep 25 16:35:50 dist-test-slave-65pf krb5kdc[15769](info): commencing operation
krb5kdc: starting...
W20260925 16:35:51.226996 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/krb5kdc/test-user.keytab.
Sep 25 16:35:51 dist-test-slave-65pf krb5kdc[15769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:35:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:35:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 16:35:51.349167 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39645
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39645
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:35:51.457007 15782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:35:51.457265 15782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:35:51.457306 15782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:35:51.461158 15782 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 16:35:51.461207 15782 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 16:35:51.461222 15782 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 16:35:51.461236 15782 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:35:51.461254 15782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:35:51.461272 15782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:35:51.461287 15782 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:35:51.461314 15782 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:35:51.466506 15782 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39645
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:35:51.467511 15782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:35:51.468456 15782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:35:51.477149 15787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:35:51.477205 15782 server_base.cc:1034] running on GCE node
W20260925 16:35:51.477149 15790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:35:51.477180 15788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:35:51.478026 15782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:35:51.479176 15782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:35:51.480362 15782 hybrid_clock.cc:648] HybridClock initialized: now 1790354151480347 us; error 28 us; skew 500 ppm
Sep 25 16:35:51 dist-test-slave-65pf krb5kdc[15769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354151, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:35:51.483088 15782 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 16:35:51.483927 15782 webserver.cc:460] Webserver started at http://127.0.98.62:45839/ using document root <none> and password file <none>
I20260925 16:35:51.484491 15782 fs_manager.cc:359] Metadata directory not provided
I20260925 16:35:51.484577 15782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:35:51.484798 15782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:35:51.486938 15782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/master-0/data/instance:
uuid: "d47abc77ad394d25b7f378cbb1ef4a60"
format_stamp: "Formatted at 2026-09-25 16:35:51 on dist-test-slave-65pf"
server_key: "ffef4d6a0e6081cdd91a92d2015c0b1f"
server_key_iv: "0ad803c70827cf49108cee6905a210a1"
server_key_version: "encryptionkey@0"
I20260925 16:35:51.487486 15782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/master-0/wal/instance:
uuid: "d47abc77ad394d25b7f378cbb1ef4a60"
format_stamp: "Formatted at 2026-09-25 16:35:51 on dist-test-slave-65pf"
server_key: "ffef4d6a0e6081cdd91a92d2015c0b1f"
server_key_iv: "0ad803c70827cf49108cee6905a210a1"
server_key_version: "encryptionkey@0"
I20260925 16:35:51.491031 15782 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 16:35:51.493248 15797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:35:51.494350 15782 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:35:51.494491 15782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/master-0/wal
uuid: "d47abc77ad394d25b7f378cbb1ef4a60"
format_stamp: "Formatted at 2026-09-25 16:35:51 on dist-test-slave-65pf"
server_key: "ffef4d6a0e6081cdd91a92d2015c0b1f"
server_key_iv: "0ad803c70827cf49108cee6905a210a1"
server_key_version: "encryptionkey@0"
I20260925 16:35:51.494616 15782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:35:51.507972 15782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:35:51.511089 15782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:35:51.511236 15782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:35:51.517565 15782 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39645
I20260925 16:35:51.517856 15849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39645 every 8 connection(s)
I20260925 16:35:51.518543 15782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:35:51.521270 15850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:35:51.524860 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 15782
I20260925 16:35:51.525007 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/master-0/wal/instance
I20260925 16:35:51.525374 392 external_mini_cluster.cc:1428] Setting key d5c56740244aabe7f330b8f82b762135
I20260925 16:35:51.529767 15850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d47abc77ad394d25b7f378cbb1ef4a60: Bootstrap starting.
I20260925 16:35:51.533181 15850 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d47abc77ad394d25b7f378cbb1ef4a60: Neither blocks nor log segments found. Creating new log.
I20260925 16:35:51.534341 15850 log.cc:824] T 00000000000000000000000000000000 P d47abc77ad394d25b7f378cbb1ef4a60: Log is configured to *not* fsync() on all Append() calls
Sep 25 16:35:51 dist-test-slave-65pf krb5kdc[15769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:35:51.537447 15850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d47abc77ad394d25b7f378cbb1ef4a60: No bootstrap required, opened a new log
I20260925 16:35:51.541564 15850 raft_consensus.cc:348] T 00000000000000000000000000000000 P d47abc77ad394d25b7f378cbb1ef4a60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d47abc77ad394d25b7f378cbb1ef4a60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39645 } }
I20260925 16:35:51.541849 15850 raft_consensus.cc:374] T 00000000000000000000000000000000 P d47abc77ad394d25b7f378cbb1ef4a60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:35:51.541949 15850 raft_consensus.cc:729] T 00000000000000000000000000000000 P d47abc77ad394d25b7f378cbb1ef4a60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d47abc77ad394d25b7f378cbb1ef4a60, State: Initialized, Role: FOLLOWER
I20260925 16:35:51.542595 15850 consensus_queue.cc:260] T 00000000000000000000000000000000 P d47abc77ad394d25b7f378cbb1ef4a60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d47abc77ad394d25b7f378cbb1ef4a60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39645 } }
I20260925 16:35:51.542763 15850 raft_consensus.cc:388] T 00000000000000000000000000000000 P d47abc77ad394d25b7f378cbb1ef4a60 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:35:51.542876 15850 raft_consensus.cc:482] T 00000000000000000000000000000000 P d47abc77ad394d25b7f378cbb1ef4a60 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:35:51.542989 15850 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d47abc77ad394d25b7f378cbb1ef4a60 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:35:51.544275 15850 raft_consensus.cc:504] T 00000000000000000000000000000000 P d47abc77ad394d25b7f378cbb1ef4a60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d47abc77ad394d25b7f378cbb1ef4a60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39645 } }
I20260925 16:35:51.544734 15850 leader_election.cc:302] T 00000000000000000000000000000000 P d47abc77ad394d25b7f378cbb1ef4a60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d47abc77ad394d25b7f378cbb1ef4a60; no voters:
I20260925 16:35:51.545096 15850 leader_election.cc:288] T 00000000000000000000000000000000 P d47abc77ad394d25b7f378cbb1ef4a60 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:35:51.545182 15853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:51.527128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40014 (local address 127.0.98.62:39645)
0925 16:35:51.527673 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0925 16:35:51.527696 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0925 16:35:51.527754 (+ 58us) server_negotiation.cc:407] Connection header received
0925 16:35:51.528503 (+ 749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:35:51.528521 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:35:51.528850 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:35:51.529175 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:35:51.530075 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:51.531785 (+ 1710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:51.533117 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:51.533698 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:51.533883 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:51.538622 (+ 4739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:35:51.538632 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:35:51.538651 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:35:51.538688 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:35:51.541498 (+ 2810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:51.542106 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:51.542112 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:51.542118 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:51.542271 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:51.542624 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:51.542628 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:51.542630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:51.543166 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:35:51.543360 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0925 16:35:51.543657 (+ 297us) server_negotiation.cc:299] Negotiation successful
0925 16:35:51.543961 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":141,"threads_started":1}
I20260925 16:35:51.545261 15855 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d47abc77ad394d25b7f378cbb1ef4a60 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:35:51.545694 15855 raft_consensus.cc:686] T 00000000000000000000000000000000 P d47abc77ad394d25b7f378cbb1ef4a60 [term 1 LEADER]: Becoming Leader. State: Replica: d47abc77ad394d25b7f378cbb1ef4a60, State: Running, Role: LEADER
I20260925 16:35:51.546025 15855 consensus_queue.cc:237] T 00000000000000000000000000000000 P d47abc77ad394d25b7f378cbb1ef4a60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d47abc77ad394d25b7f378cbb1ef4a60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39645 } }
I20260925 16:35:51.546566 15850 sys_catalog.cc:564] T 00000000000000000000000000000000 P d47abc77ad394d25b7f378cbb1ef4a60 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:35:51.548053 15856 sys_catalog.cc:455] T 00000000000000000000000000000000 P d47abc77ad394d25b7f378cbb1ef4a60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d47abc77ad394d25b7f378cbb1ef4a60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d47abc77ad394d25b7f378cbb1ef4a60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39645 } } }
I20260925 16:35:51.548305 15856 sys_catalog.cc:458] T 00000000000000000000000000000000 P d47abc77ad394d25b7f378cbb1ef4a60 [sys.catalog]: This master's current role is: LEADER
I20260925 16:35:51.549117 15857 sys_catalog.cc:455] T 00000000000000000000000000000000 P d47abc77ad394d25b7f378cbb1ef4a60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d47abc77ad394d25b7f378cbb1ef4a60. Latest consensus state: current_term: 1 leader_uuid: "d47abc77ad394d25b7f378cbb1ef4a60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d47abc77ad394d25b7f378cbb1ef4a60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39645 } } }
I20260925 16:35:51.549274 15857 sys_catalog.cc:458] T 00000000000000000000000000000000 P d47abc77ad394d25b7f378cbb1ef4a60 [sys.catalog]: This master's current role is: LEADER
W20260925 16:35:51.552465 15870 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d47abc77ad394d25b7f378cbb1ef4a60: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 16:35:51.552585 15870 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 16:35:51.552707 15869 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:35:51.553687 15869 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:35:51.560266 15869 catalog_manager.cc:1347] Generated new cluster ID: b54e38f54c674aca97a7b343c71948b4
I20260925 16:35:51.560331 15869 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:35:51.573964 15869 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:35:51.575380 15869 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:35:51.587404 15869 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d47abc77ad394d25b7f378cbb1ef4a60: Generated new TSK 0
I20260925 16:35:51.588255 15869 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 16:35:51.626372 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39645
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:35:51.733364 15876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:35:51.733603 15876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:35:51.733642 15876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:35:51.737574 15876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:35:51.737628 15876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:35:51.737682 15876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 16:35:51.742707 15876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:35:51.743685 15876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:35:51.744634 15876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:35:51.753873 15882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:35:51.753912 15884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:35:51.753898 15881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:35:51.754195 15876 server_base.cc:1034] running on GCE node
I20260925 16:35:51.754640 15876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:35:51.755198 15876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:35:51.756349 15876 hybrid_clock.cc:648] HybridClock initialized: now 1790354151756334 us; error 27 us; skew 500 ppm
Sep 25 16:35:51 dist-test-slave-65pf krb5kdc[15769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354151, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:35:51.759429 15876 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 16:35:51.760279 15876 webserver.cc:460] Webserver started at http://127.0.98.1:45121/ using document root <none> and password file <none>
I20260925 16:35:51.760815 15876 fs_manager.cc:359] Metadata directory not provided
I20260925 16:35:51.760871 15876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:35:51.761080 15876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:35:51.763096 15876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-0/data/instance:
uuid: "fd2d720c99f84f098a1054646aedd2a9"
format_stamp: "Formatted at 2026-09-25 16:35:51 on dist-test-slave-65pf"
server_key: "600604f57e929abf76e71ef83692abb6"
server_key_iv: "cb60f45afaf6022379f1a9fedb1ace19"
server_key_version: "encryptionkey@0"
I20260925 16:35:51.763638 15876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-0/wal/instance:
uuid: "fd2d720c99f84f098a1054646aedd2a9"
format_stamp: "Formatted at 2026-09-25 16:35:51 on dist-test-slave-65pf"
server_key: "600604f57e929abf76e71ef83692abb6"
server_key_iv: "cb60f45afaf6022379f1a9fedb1ace19"
server_key_version: "encryptionkey@0"
I20260925 16:35:51.767155 15876 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 16:35:51.769542 15891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:35:51.770771 15876 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:35:51.770907 15876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-0/wal
uuid: "fd2d720c99f84f098a1054646aedd2a9"
format_stamp: "Formatted at 2026-09-25 16:35:51 on dist-test-slave-65pf"
server_key: "600604f57e929abf76e71ef83692abb6"
server_key_iv: "cb60f45afaf6022379f1a9fedb1ace19"
server_key_version: "encryptionkey@0"
I20260925 16:35:51.771003 15876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:35:51.785549 15876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:35:51.788362 15876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:35:51.788549 15876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:35:51.789090 15876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:35:51.789918 15876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:35:51.789963 15876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:35:51.790000 15876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:35:51.790050 15876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:35:51.799023 15876 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37719
I20260925 16:35:51.799089 16004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37719 every 8 connection(s)
I20260925 16:35:51.799878 15876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
I20260925 16:35:51.802459 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 15876
I20260925 16:35:51.802619 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-0/wal/instance
I20260925 16:35:51.802944 392 external_mini_cluster.cc:1428] Setting key 4a2c2edf54b8b0955ccd34d21cb8819c
Sep 25 16:35:51 dist-test-slave-65pf krb5kdc[15769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354151, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 16:35:51.814479 15853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:51.801582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33039 (local address 127.0.98.62:39645)
0925 16:35:51.801736 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 16:35:51.801741 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:35:51.802528 (+ 787us) server_negotiation.cc:407] Connection header received
0925 16:35:51.804102 (+ 1574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:35:51.804109 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:35:51.804191 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:35:51.804294 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:35:51.805599 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:51.806353 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:51.808136 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:51.808353 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:51.808469 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:51.811088 (+ 2619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:35:51.811092 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:35:51.811094 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:35:51.811116 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:35:51.812455 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:51.812941 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:51.812949 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:51.812950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:51.813019 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:51.813346 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:51.813350 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:51.813351 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:51.813525 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:35:51.813627 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 16:35:51.814190 (+ 563us) server_negotiation.cc:299] Negotiation successful
0925 16:35:51.814309 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 16:35:51.815435 16007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:51.801974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39645 (local address 127.0.98.1:33039)
0925 16:35:51.802392 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:51.802432 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 16:35:51.803808 (+ 1376us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:35:51.804498 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:35:51.804509 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:35:51.804940 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:35:51.805443 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:51.805456 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:51.806476 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:51.806481 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:51.808020 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:51.808034 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:51.808510 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:51.808518 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:51.808757 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:51.809419 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:35:51.809435 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:35:51.810933 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:35:51.812570 (+ 1637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:35:51.812576 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:35:51.812589 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:35:51.812832 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:35:51.813113 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:35:51.813116 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:35:51.813118 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:35:51.813248 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:35:51.813627 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:35:51.813633 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:35:51.813961 (+ 328us) client_negotiation.cc:769] Sending connection context
0925 16:35:51.814134 (+ 173us) client_negotiation.cc:240] Negotiation successful
0925 16:35:51.814448 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":101,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260925 16:35:51.816628 16005 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39645
I20260925 16:35:51.816880 16005 heartbeater.cc:464] Registering TS with master...
I20260925 16:35:51.817385 16005 heartbeater.cc:511] Master 127.0.98.62:39645 requested a full tablet report, sending...
I20260925 16:35:51.819000 15814 ts_manager.cc:194] Registered new tserver with Master: fd2d720c99f84f098a1054646aedd2a9 (127.0.98.1:37719)
I20260925 16:35:51.820717 15814 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33039
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 16:35:51.867440 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39645
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:35:51.975466 16010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:35:51.975724 16010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:35:51.975767 16010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:35:51.979730 16010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:35:51.979792 16010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:35:51.979851 16010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 16:35:51.984956 16010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:35:51.985971 16010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:35:51.987083 16010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:35:51.996615 16016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:35:51.996631 16018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:35:51.996817 16015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:35:51.997030 16010 server_base.cc:1034] running on GCE node
I20260925 16:35:51.997453 16010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:35:51.998088 16010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:35:51.999286 16010 hybrid_clock.cc:648] HybridClock initialized: now 1790354151999258 us; error 40 us; skew 500 ppm
Sep 25 16:35:51 dist-test-slave-65pf krb5kdc[15769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354151, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:35:52.002102 16010 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 16:35:52.003069 16010 webserver.cc:460] Webserver started at http://127.0.98.2:36163/ using document root <none> and password file <none>
I20260925 16:35:52.003665 16010 fs_manager.cc:359] Metadata directory not provided
I20260925 16:35:52.003752 16010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:35:52.003993 16010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:35:52.006052 16010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-1/data/instance:
uuid: "a114478b9c5b4e14a0de2caf167fe07b"
format_stamp: "Formatted at 2026-09-25 16:35:51 on dist-test-slave-65pf"
server_key: "fdb9f4d9226fa4df152bf958bbdfe358"
server_key_iv: "daf8c7d45243c5ed6221e8d143baadf0"
server_key_version: "encryptionkey@0"
I20260925 16:35:52.006659 16010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a114478b9c5b4e14a0de2caf167fe07b"
format_stamp: "Formatted at 2026-09-25 16:35:51 on dist-test-slave-65pf"
server_key: "fdb9f4d9226fa4df152bf958bbdfe358"
server_key_iv: "daf8c7d45243c5ed6221e8d143baadf0"
server_key_version: "encryptionkey@0"
I20260925 16:35:52.010337 16010 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 16:35:52.012945 16025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:35:52.014209 16010 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:35:52.014302 16010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-1/wal
uuid: "a114478b9c5b4e14a0de2caf167fe07b"
format_stamp: "Formatted at 2026-09-25 16:35:51 on dist-test-slave-65pf"
server_key: "fdb9f4d9226fa4df152bf958bbdfe358"
server_key_iv: "daf8c7d45243c5ed6221e8d143baadf0"
server_key_version: "encryptionkey@0"
I20260925 16:35:52.014426 16010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:35:52.053112 16010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:35:52.056424 16010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:35:52.056581 16010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:35:52.057207 16010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:35:52.058017 16010 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:35:52.058064 16010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:35:52.058104 16010 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:35:52.058120 16010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:35:52.066371 16010 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43913
I20260925 16:35:52.066462 16138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43913 every 8 connection(s)
I20260925 16:35:52.067278 16010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
I20260925 16:35:52.074002 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 16010
I20260925 16:35:52.074132 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-1/wal/instance
I20260925 16:35:52.074424 392 external_mini_cluster.cc:1428] Setting key d793def308458ef53f01d37291f5c972
Sep 25 16:35:52 dist-test-slave-65pf krb5kdc[15769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354151, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:35:52.080878 15853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:52.068989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56707 (local address 127.0.98.62:39645)
0925 16:35:52.069120 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 16:35:52.069123 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:35:52.069867 (+ 744us) server_negotiation.cc:407] Connection header received
0925 16:35:52.071096 (+ 1229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:35:52.071099 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:35:52.071147 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:35:52.071223 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:35:52.072305 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:52.072816 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:52.074028 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:52.074339 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:52.074451 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:52.076998 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:35:52.077002 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:35:52.077005 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:35:52.077028 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:35:52.078780 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:52.079243 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:52.079249 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:52.079254 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:52.079345 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:52.079655 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:52.079661 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:52.079666 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:52.079913 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:35:52.080010 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 16:35:52.080550 (+ 540us) server_negotiation.cc:299] Negotiation successful
0925 16:35:52.080683 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 16:35:52.081871 16141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:52.069295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39645 (local address 127.0.98.2:56707)
0925 16:35:52.069706 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:52.069747 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 16:35:52.070889 (+ 1142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:35:52.071386 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:35:52.071392 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:35:52.071728 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:35:52.072181 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:52.072194 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:52.072932 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:52.072934 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:52.073914 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:52.073923 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:52.074487 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:52.074491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:52.074701 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:52.075302 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:35:52.075318 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:35:52.076846 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:35:52.078887 (+ 2041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:35:52.078892 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:35:52.078903 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:35:52.079146 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:35:52.079443 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:35:52.079445 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:35:52.079446 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:35:52.079569 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:35:52.080016 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:35:52.080020 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:35:52.080282 (+ 262us) client_negotiation.cc:769] Sending connection context
0925 16:35:52.080466 (+ 184us) client_negotiation.cc:240] Negotiation successful
0925 16:35:52.080811 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":100,"threads_started":1}
I20260925 16:35:52.083133 16139 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39645
I20260925 16:35:52.083379 16139 heartbeater.cc:464] Registering TS with master...
I20260925 16:35:52.083853 16139 heartbeater.cc:511] Master 127.0.98.62:39645 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260925 16:35:52.084892 15814 ts_manager.cc:194] Registered new tserver with Master: a114478b9c5b4e14a0de2caf167fe07b (127.0.98.2:43913)
I20260925 16:35:52.085551 15814 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56707
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 16:35:52.161755 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39645
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:35:52.272074 16144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:35:52.272342 16144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:35:52.272390 16144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:35:52.276443 16144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:35:52.276512 16144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:35:52.276575 16144 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 16:35:52.281965 16144 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:35:52.283010 16144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:35:52.284008 16144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:35:52.293586 16150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:35:52.293579 16149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:35:52.293589 16152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:35:52.294055 16144 server_base.cc:1034] running on GCE node
I20260925 16:35:52.294490 16144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:35:52.295162 16144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:35:52.296329 16144 hybrid_clock.cc:648] HybridClock initialized: now 1790354152296317 us; error 28 us; skew 500 ppm
Sep 25 16:35:52 dist-test-slave-65pf krb5kdc[15769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:35:52.299410 16144 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 16:35:52.300285 16144 webserver.cc:460] Webserver started at http://127.0.98.3:33447/ using document root <none> and password file <none>
I20260925 16:35:52.300814 16144 fs_manager.cc:359] Metadata directory not provided
I20260925 16:35:52.300875 16144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:35:52.301088 16144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:35:52.303143 16144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-2/data/instance:
uuid: "95b51fa015394f2e892650b4f9e1bfd2"
format_stamp: "Formatted at 2026-09-25 16:35:52 on dist-test-slave-65pf"
server_key: "1d2c2f1d0f4729f800dcafd2508a7234"
server_key_iv: "072658af143c19ca82e2fda2a4849fd3"
server_key_version: "encryptionkey@0"
I20260925 16:35:52.303650 16144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-2/wal/instance:
uuid: "95b51fa015394f2e892650b4f9e1bfd2"
format_stamp: "Formatted at 2026-09-25 16:35:52 on dist-test-slave-65pf"
server_key: "1d2c2f1d0f4729f800dcafd2508a7234"
server_key_iv: "072658af143c19ca82e2fda2a4849fd3"
server_key_version: "encryptionkey@0"
I20260925 16:35:52.307243 16144 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 16:35:52.309526 16159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:35:52.310799 16144 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:35:52.310923 16144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-2/wal
uuid: "95b51fa015394f2e892650b4f9e1bfd2"
format_stamp: "Formatted at 2026-09-25 16:35:52 on dist-test-slave-65pf"
server_key: "1d2c2f1d0f4729f800dcafd2508a7234"
server_key_iv: "072658af143c19ca82e2fda2a4849fd3"
server_key_version: "encryptionkey@0"
I20260925 16:35:52.311031 16144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:35:52.322220 16144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:35:52.325021 16144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:35:52.325212 16144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:35:52.325779 16144 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:35:52.326556 16144 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:35:52.326659 16144 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:35:52.326730 16144 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:35:52.326776 16144 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:35:52.336030 16144 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37195
I20260925 16:35:52.336047 16272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37195 every 8 connection(s)
I20260925 16:35:52.336908 16144 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
I20260925 16:35:52.337855 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 16144
I20260925 16:35:52.337988 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-2/wal/instance
I20260925 16:35:52.338297 392 external_mini_cluster.cc:1428] Setting key 37060537256d03d22af685f87aa0581e
Sep 25 16:35:52 dist-test-slave-65pf krb5kdc[15769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:35:52.349915 15853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:52.338800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43015 (local address 127.0.98.62:39645)
0925 16:35:52.338946 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 16:35:52.338950 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:35:52.339872 (+ 922us) server_negotiation.cc:407] Connection header received
0925 16:35:52.341169 (+ 1297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:35:52.341172 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:35:52.341227 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:35:52.341302 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:35:52.342405 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:52.343005 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:52.344095 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:52.344328 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:52.344432 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:52.346800 (+ 2368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:35:52.346803 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:35:52.346805 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:35:52.346830 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:35:52.348230 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:52.348692 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:52.348695 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:52.348696 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:52.348771 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:52.349048 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:52.349050 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:52.349051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:52.349209 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:35:52.349280 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 16:35:52.349699 (+ 419us) server_negotiation.cc:299] Negotiation successful
0925 16:35:52.349791 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 16:35:52.350744 16275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:52.339252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39645 (local address 127.0.98.3:43015)
0925 16:35:52.339740 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:52.339782 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 16:35:52.340976 (+ 1194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:35:52.341469 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:35:52.341476 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:35:52.341815 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:35:52.342267 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:52.342279 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:52.343121 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:52.343124 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:52.343978 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:52.343986 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:52.344463 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:52.344467 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:52.344625 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:52.345119 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:35:52.345133 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:35:52.346566 (+ 1433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:35:52.348350 (+ 1784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:35:52.348354 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:35:52.348366 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:35:52.348611 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:35:52.348852 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:35:52.348854 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:35:52.348856 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:35:52.348977 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:35:52.349288 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:35:52.349293 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:35:52.349512 (+ 219us) client_negotiation.cc:769] Sending connection context
0925 16:35:52.349659 (+ 147us) client_negotiation.cc:240] Negotiation successful
0925 16:35:52.349894 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":107,"threads_started":1}
I20260925 16:35:52.351982 16273 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39645
I20260925 16:35:52.352216 16273 heartbeater.cc:464] Registering TS with master...
I20260925 16:35:52.352639 16273 heartbeater.cc:511] Master 127.0.98.62:39645 requested a full tablet report, sending...
I20260925 16:35:52.353535 15814 ts_manager.cc:194] Registered new tserver with Master: 95b51fa015394f2e892650b4f9e1bfd2 (127.0.98.3:37195)
I20260925 16:35:52.354111 15814 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43015
I20260925 16:35:52.363711 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 16:35:52.372445 15853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:52.365592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40026 (local address 127.0.98.62:39645)
0925 16:35:52.365743 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 16:35:52.365749 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:35:52.365874 (+ 125us) server_negotiation.cc:407] Connection header received
0925 16:35:52.365973 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:35:52.365975 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:35:52.366013 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:35:52.366077 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:35:52.366823 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:52.367335 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:52.368181 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:52.368404 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:52.368485 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:52.369712 (+ 1227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:35:52.369715 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:35:52.369717 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:35:52.369740 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:35:52.371040 (+ 1300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:52.371401 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:52.371403 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:52.371404 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:52.371471 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:35:52.371679 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:35:52.371680 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:35:52.371681 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:35:52.371905 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:35:52.372003 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 16:35:52.372173 (+ 170us) server_negotiation.cc:299] Negotiation successful
0925 16:35:52.372260 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 16:35:52.375340 15814 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40026:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 16:35:52.378299 15814 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:35:52.393492 16284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:52.386484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37195 (local address 127.0.0.1:54228)
0925 16:35:52.386845 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:52.386897 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 16:35:52.387027 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:35:52.388475 (+ 1448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:35:52.388479 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:35:52.388508 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:35:52.388674 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:52.388680 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:52.390071 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:52.390075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:52.391664 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:52.391674 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:52.392947 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:52.392950 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:52.393053 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:52.393114 (+ 61us) client_negotiation.cc:769] Sending connection context
0925 16:35:52.393214 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 16:35:52.393322 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":45,"spinlock_wait_cycles":4736,"thread_start_us":70,"threads_started":1}
I20260925 16:35:52.393782 16288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:52.387279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47362 (local address 127.0.98.1:37719)
0925 16:35:52.387943 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0925 16:35:52.387952 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 16:35:52.387976 (+ 24us) server_negotiation.cc:407] Connection header received
0925 16:35:52.388050 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:35:52.388054 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:35:52.388207 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:35:52.388376 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:35:52.388690 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:52.390278 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:52.391900 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:52.392690 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:52.392783 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:52.392903 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 16:35:52.393557 (+ 654us) server_negotiation.cc:299] Negotiation successful
0925 16:35:52.393650 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":83,"threads_started":1}
I20260925 16:35:52.393941 16285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:52.386645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54228 (local address 127.0.98.3:37195)
0925 16:35:52.388006 (+ 1361us) server_negotiation.cc:206] Beginning negotiation
0925 16:35:52.388019 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 16:35:52.388041 (+ 22us) server_negotiation.cc:407] Connection header received
0925 16:35:52.388114 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:35:52.388120 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:35:52.388323 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:35:52.388578 (+ 255us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:35:52.388772 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:52.389930 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:52.391786 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:52.392602 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:52.392712 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:52.392887 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 16:35:52.393691 (+ 804us) server_negotiation.cc:299] Negotiation successful
0925 16:35:52.393795 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1260,"thread_start_us":162,"threads_started":1}
I20260925 16:35:52.394003 16287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:52.386711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37719 (local address 127.0.0.1:47362)
0925 16:35:52.387505 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:52.387521 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:35:52.387751 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:35:52.388364 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:35:52.388371 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:35:52.388393 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:35:52.388553 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:52.388564 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:52.390388 (+ 1824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:52.390393 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:52.391801 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:52.391806 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:52.392959 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:52.392962 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:52.393103 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:52.393126 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 16:35:52.393398 (+ 272us) client_negotiation.cc:240] Negotiation successful
0925 16:35:52.393466 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":737,"mutex_wait_us":209,"thread_start_us":119,"threads_started":1}
I20260925 16:35:52.394742 16286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:52.387208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43913 (local address 127.0.0.1:46176)
0925 16:35:52.387645 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:52.387660 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:35:52.387730 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:35:52.388475 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:35:52.388479 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:35:52.388509 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:35:52.388640 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:52.388646 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:52.390166 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:52.390171 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:52.392672 (+ 2501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:52.392684 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:52.394342 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:52.394346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:52.394438 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:52.394459 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 16:35:52.394532 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 16:35:52.394619 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"mutex_wait_us":1281,"spinlock_wait_cycles":5632,"thread_start_us":95,"threads_started":1}
I20260925 16:35:52.394850 16289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:52.387397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46176 (local address 127.0.98.2:43913)
0925 16:35:52.388006 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0925 16:35:52.388020 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 16:35:52.388041 (+ 21us) server_negotiation.cc:407] Connection header received
0925 16:35:52.388114 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:35:52.388120 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:35:52.388315 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:35:52.388778 (+ 463us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:35:52.388958 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:52.390028 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:52.392797 (+ 2769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:52.394221 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:52.394324 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:52.394414 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 16:35:52.394601 (+ 187us) server_negotiation.cc:299] Negotiation successful
0925 16:35:52.394693 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":98,"threads_started":1}
I20260925 16:35:52.397269 16207 tablet_service.cc:1467] Processing CreateTablet for tablet 57c3d9c330f64fdeab1886abf778dabc (DEFAULT_TABLE table=test-table [id=77692f05043f4f16a97b53f2e1d59d65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:35:52.397353 16073 tablet_service.cc:1467] Processing CreateTablet for tablet 57c3d9c330f64fdeab1886abf778dabc (DEFAULT_TABLE table=test-table [id=77692f05043f4f16a97b53f2e1d59d65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:35:52.397832 15939 tablet_service.cc:1467] Processing CreateTablet for tablet 57c3d9c330f64fdeab1886abf778dabc (DEFAULT_TABLE table=test-table [id=77692f05043f4f16a97b53f2e1d59d65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:35:52.398406 16207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57c3d9c330f64fdeab1886abf778dabc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:35:52.398401 16073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57c3d9c330f64fdeab1886abf778dabc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:35:52.398705 15939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57c3d9c330f64fdeab1886abf778dabc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:35:52.404500 16290 tablet_bootstrap.cc:492] T 57c3d9c330f64fdeab1886abf778dabc P 95b51fa015394f2e892650b4f9e1bfd2: Bootstrap starting.
I20260925 16:35:52.405157 16291 tablet_bootstrap.cc:492] T 57c3d9c330f64fdeab1886abf778dabc P fd2d720c99f84f098a1054646aedd2a9: Bootstrap starting.
I20260925 16:35:52.405566 16292 tablet_bootstrap.cc:492] T 57c3d9c330f64fdeab1886abf778dabc P a114478b9c5b4e14a0de2caf167fe07b: Bootstrap starting.
I20260925 16:35:52.408277 16291 tablet_bootstrap.cc:654] T 57c3d9c330f64fdeab1886abf778dabc P fd2d720c99f84f098a1054646aedd2a9: Neither blocks nor log segments found. Creating new log.
I20260925 16:35:52.408268 16292 tablet_bootstrap.cc:654] T 57c3d9c330f64fdeab1886abf778dabc P a114478b9c5b4e14a0de2caf167fe07b: Neither blocks nor log segments found. Creating new log.
I20260925 16:35:52.408268 16290 tablet_bootstrap.cc:654] T 57c3d9c330f64fdeab1886abf778dabc P 95b51fa015394f2e892650b4f9e1bfd2: Neither blocks nor log segments found. Creating new log.
I20260925 16:35:52.408996 16291 log.cc:824] T 57c3d9c330f64fdeab1886abf778dabc P fd2d720c99f84f098a1054646aedd2a9: Log is configured to *not* fsync() on all Append() calls
I20260925 16:35:52.409122 16292 log.cc:824] T 57c3d9c330f64fdeab1886abf778dabc P a114478b9c5b4e14a0de2caf167fe07b: Log is configured to *not* fsync() on all Append() calls
I20260925 16:35:52.409122 16290 log.cc:824] T 57c3d9c330f64fdeab1886abf778dabc P 95b51fa015394f2e892650b4f9e1bfd2: Log is configured to *not* fsync() on all Append() calls
I20260925 16:35:52.411077 16290 tablet_bootstrap.cc:492] T 57c3d9c330f64fdeab1886abf778dabc P 95b51fa015394f2e892650b4f9e1bfd2: No bootstrap required, opened a new log
I20260925 16:35:52.411077 16291 tablet_bootstrap.cc:492] T 57c3d9c330f64fdeab1886abf778dabc P fd2d720c99f84f098a1054646aedd2a9: No bootstrap required, opened a new log
I20260925 16:35:52.411278 16291 ts_tablet_manager.cc:1397] T 57c3d9c330f64fdeab1886abf778dabc P fd2d720c99f84f098a1054646aedd2a9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 16:35:52.411269 16290 ts_tablet_manager.cc:1397] T 57c3d9c330f64fdeab1886abf778dabc P 95b51fa015394f2e892650b4f9e1bfd2: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 16:35:52.411854 16292 tablet_bootstrap.cc:492] T 57c3d9c330f64fdeab1886abf778dabc P a114478b9c5b4e14a0de2caf167fe07b: No bootstrap required, opened a new log
I20260925 16:35:52.412021 16292 ts_tablet_manager.cc:1397] T 57c3d9c330f64fdeab1886abf778dabc P a114478b9c5b4e14a0de2caf167fe07b: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260925 16:35:52.415086 16292 raft_consensus.cc:348] T 57c3d9c330f64fdeab1886abf778dabc P a114478b9c5b4e14a0de2caf167fe07b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95b51fa015394f2e892650b4f9e1bfd2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } } peers { permanent_uuid: "fd2d720c99f84f098a1054646aedd2a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37719 } } peers { permanent_uuid: "a114478b9c5b4e14a0de2caf167fe07b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43913 } }
I20260925 16:35:52.415385 16292 raft_consensus.cc:374] T 57c3d9c330f64fdeab1886abf778dabc P a114478b9c5b4e14a0de2caf167fe07b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:35:52.415244 16290 raft_consensus.cc:348] T 57c3d9c330f64fdeab1886abf778dabc P 95b51fa015394f2e892650b4f9e1bfd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95b51fa015394f2e892650b4f9e1bfd2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } } peers { permanent_uuid: "fd2d720c99f84f098a1054646aedd2a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37719 } } peers { permanent_uuid: "a114478b9c5b4e14a0de2caf167fe07b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43913 } }
I20260925 16:35:52.415468 16292 raft_consensus.cc:729] T 57c3d9c330f64fdeab1886abf778dabc P a114478b9c5b4e14a0de2caf167fe07b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a114478b9c5b4e14a0de2caf167fe07b, State: Initialized, Role: FOLLOWER
I20260925 16:35:52.415540 16290 raft_consensus.cc:374] T 57c3d9c330f64fdeab1886abf778dabc P 95b51fa015394f2e892650b4f9e1bfd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:35:52.415601 16290 raft_consensus.cc:729] T 57c3d9c330f64fdeab1886abf778dabc P 95b51fa015394f2e892650b4f9e1bfd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95b51fa015394f2e892650b4f9e1bfd2, State: Initialized, Role: FOLLOWER
I20260925 16:35:52.415704 16291 raft_consensus.cc:348] T 57c3d9c330f64fdeab1886abf778dabc P fd2d720c99f84f098a1054646aedd2a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95b51fa015394f2e892650b4f9e1bfd2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } } peers { permanent_uuid: "fd2d720c99f84f098a1054646aedd2a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37719 } } peers { permanent_uuid: "a114478b9c5b4e14a0de2caf167fe07b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43913 } }
I20260925 16:35:52.415935 16291 raft_consensus.cc:374] T 57c3d9c330f64fdeab1886abf778dabc P fd2d720c99f84f098a1054646aedd2a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:35:52.415992 16291 raft_consensus.cc:729] T 57c3d9c330f64fdeab1886abf778dabc P fd2d720c99f84f098a1054646aedd2a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd2d720c99f84f098a1054646aedd2a9, State: Initialized, Role: FOLLOWER
I20260925 16:35:52.415964 16292 consensus_queue.cc:260] T 57c3d9c330f64fdeab1886abf778dabc P a114478b9c5b4e14a0de2caf167fe07b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95b51fa015394f2e892650b4f9e1bfd2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } } peers { permanent_uuid: "fd2d720c99f84f098a1054646aedd2a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37719 } } peers { permanent_uuid: "a114478b9c5b4e14a0de2caf167fe07b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43913 } }
I20260925 16:35:52.416007 16290 consensus_queue.cc:260] T 57c3d9c330f64fdeab1886abf778dabc P 95b51fa015394f2e892650b4f9e1bfd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95b51fa015394f2e892650b4f9e1bfd2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } } peers { permanent_uuid: "fd2d720c99f84f098a1054646aedd2a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37719 } } peers { permanent_uuid: "a114478b9c5b4e14a0de2caf167fe07b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43913 } }
I20260925 16:35:52.416406 16291 consensus_queue.cc:260] T 57c3d9c330f64fdeab1886abf778dabc P fd2d720c99f84f098a1054646aedd2a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95b51fa015394f2e892650b4f9e1bfd2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } } peers { permanent_uuid: "fd2d720c99f84f098a1054646aedd2a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37719 } } peers { permanent_uuid: "a114478b9c5b4e14a0de2caf167fe07b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43913 } }
I20260925 16:35:52.416891 16139 heartbeater.cc:503] Master 127.0.98.62:39645 was elected leader, sending a full tablet report...
I20260925 16:35:52.416999 16290 ts_tablet_manager.cc:1428] T 57c3d9c330f64fdeab1886abf778dabc P 95b51fa015394f2e892650b4f9e1bfd2: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 16:35:52.417052 16292 ts_tablet_manager.cc:1428] T 57c3d9c330f64fdeab1886abf778dabc P a114478b9c5b4e14a0de2caf167fe07b: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 16:35:52.417200 16291 ts_tablet_manager.cc:1428] T 57c3d9c330f64fdeab1886abf778dabc P fd2d720c99f84f098a1054646aedd2a9: Time spent starting tablet: real 0.006s user 0.000s sys 0.005s
I20260925 16:35:52.417480 16005 heartbeater.cc:503] Master 127.0.98.62:39645 was elected leader, sending a full tablet report...
I20260925 16:35:52.417937 16273 heartbeater.cc:503] Master 127.0.98.62:39645 was elected leader, sending a full tablet report...
I20260925 16:35:52.550529 16296 raft_consensus.cc:482] T 57c3d9c330f64fdeab1886abf778dabc P a114478b9c5b4e14a0de2caf167fe07b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:35:52.550791 16296 raft_consensus.cc:504] T 57c3d9c330f64fdeab1886abf778dabc P a114478b9c5b4e14a0de2caf167fe07b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95b51fa015394f2e892650b4f9e1bfd2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } } peers { permanent_uuid: "fd2d720c99f84f098a1054646aedd2a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37719 } } peers { permanent_uuid: "a114478b9c5b4e14a0de2caf167fe07b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43913 } }
W20260925 16:35:52.551358 16006 tablet.cc:2369] T 57c3d9c330f64fdeab1886abf778dabc P fd2d720c99f84f098a1054646aedd2a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:35:52.552058 16296 leader_election.cc:288] T 57c3d9c330f64fdeab1886abf778dabc P a114478b9c5b4e14a0de2caf167fe07b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 95b51fa015394f2e892650b4f9e1bfd2 (127.0.98.3:37195), fd2d720c99f84f098a1054646aedd2a9 (127.0.98.1:37719)
I20260925 16:35:52.556941 16288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:52.552532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57391 (local address 127.0.98.1:37719)
0925 16:35:52.552641 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0925 16:35:52.552645 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:35:52.552877 (+ 232us) server_negotiation.cc:407] Connection header received
0925 16:35:52.552968 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:35:52.552970 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:35:52.553008 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:35:52.553086 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:35:52.553311 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:52.554011 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:52.555634 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:52.556280 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:52.556406 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:52.556468 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 16:35:52.556742 (+ 274us) server_negotiation.cc:299] Negotiation successful
0925 16:35:52.556812 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 16:35:52.556951 16285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:52.552458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49731 (local address 127.0.98.3:37195)
0925 16:35:52.552589 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 16:35:52.552593 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:35:52.552611 (+ 18us) server_negotiation.cc:407] Connection header received
0925 16:35:52.552670 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:35:52.552674 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:35:52.552728 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:35:52.552802 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:35:52.553132 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:52.553920 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:52.555634 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:52.556280 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:52.556405 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:52.556468 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 16:35:52.556742 (+ 274us) server_negotiation.cc:299] Negotiation successful
0925 16:35:52.556812 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 16:35:52.557035 16141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:52.552347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37195 (local address 127.0.98.2:49731)
0925 16:35:52.552446 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:52.552462 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:35:52.552558 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:35:52.552849 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:35:52.552852 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:35:52.552872 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:35:52.553009 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:52.553014 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:52.554025 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:52.554028 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:52.555466 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:52.555473 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:52.556420 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:52.556424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:52.556596 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:52.556634 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 16:35:52.556715 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 16:35:52.556877 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24,"mutex_wait_us":125}
I20260925 16:35:52.557045 16299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:52.552347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37719 (local address 127.0.98.2:57391)
0925 16:35:52.552755 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:52.552766 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 16:35:52.552885 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:35:52.553103 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:35:52.553108 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:35:52.553128 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:35:52.553217 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:52.553221 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:52.554107 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:52.554109 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:52.555455 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:35:52.555466 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:35:52.556420 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:35:52.556424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:35:52.556592 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:52.556643 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 16:35:52.556715 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 16:35:52.556877 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":319,"spinlock_wait_cycles":6528,"thread_start_us":97,"threads_started":1}
I20260925 16:35:52.557783 16227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57c3d9c330f64fdeab1886abf778dabc" candidate_uuid: "a114478b9c5b4e14a0de2caf167fe07b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95b51fa015394f2e892650b4f9e1bfd2" is_pre_election: true
I20260925 16:35:52.557807 15959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57c3d9c330f64fdeab1886abf778dabc" candidate_uuid: "a114478b9c5b4e14a0de2caf167fe07b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd2d720c99f84f098a1054646aedd2a9" is_pre_election: true
I20260925 16:35:52.558086 16227 raft_consensus.cc:2454] T 57c3d9c330f64fdeab1886abf778dabc P 95b51fa015394f2e892650b4f9e1bfd2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a114478b9c5b4e14a0de2caf167fe07b in term 0.
I20260925 16:35:52.558106 15959 raft_consensus.cc:2454] T 57c3d9c330f64fdeab1886abf778dabc P fd2d720c99f84f098a1054646aedd2a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a114478b9c5b4e14a0de2caf167fe07b in term 0.
I20260925 16:35:52.558619 16027 leader_election.cc:302] T 57c3d9c330f64fdeab1886abf778dabc P a114478b9c5b4e14a0de2caf167fe07b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a114478b9c5b4e14a0de2caf167fe07b, fd2d720c99f84f098a1054646aedd2a9; no voters:
I20260925 16:35:52.558919 16296 raft_consensus.cc:2781] T 57c3d9c330f64fdeab1886abf778dabc P a114478b9c5b4e14a0de2caf167fe07b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:35:52.559006 16296 raft_consensus.cc:482] T 57c3d9c330f64fdeab1886abf778dabc P a114478b9c5b4e14a0de2caf167fe07b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:35:52.559062 16296 raft_consensus.cc:3037] T 57c3d9c330f64fdeab1886abf778dabc P a114478b9c5b4e14a0de2caf167fe07b [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:35:52.560062 16296 raft_consensus.cc:504] T 57c3d9c330f64fdeab1886abf778dabc P a114478b9c5b4e14a0de2caf167fe07b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95b51fa015394f2e892650b4f9e1bfd2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } } peers { permanent_uuid: "fd2d720c99f84f098a1054646aedd2a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37719 } } peers { permanent_uuid: "a114478b9c5b4e14a0de2caf167fe07b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43913 } }
I20260925 16:35:52.560406 16296 leader_election.cc:288] T 57c3d9c330f64fdeab1886abf778dabc P a114478b9c5b4e14a0de2caf167fe07b [CANDIDATE]: Term 1 election: Requested vote from peers 95b51fa015394f2e892650b4f9e1bfd2 (127.0.98.3:37195), fd2d720c99f84f098a1054646aedd2a9 (127.0.98.1:37719)
I20260925 16:35:52.560742 16227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57c3d9c330f64fdeab1886abf778dabc" candidate_uuid: "a114478b9c5b4e14a0de2caf167fe07b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95b51fa015394f2e892650b4f9e1bfd2"
I20260925 16:35:52.560865 16227 raft_consensus.cc:3037] T 57c3d9c330f64fdeab1886abf778dabc P 95b51fa015394f2e892650b4f9e1bfd2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:35:52.561910 16227 raft_consensus.cc:2454] T 57c3d9c330f64fdeab1886abf778dabc P 95b51fa015394f2e892650b4f9e1bfd2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a114478b9c5b4e14a0de2caf167fe07b in term 1.
I20260925 16:35:52.562188 15959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57c3d9c330f64fdeab1886abf778dabc" candidate_uuid: "a114478b9c5b4e14a0de2caf167fe07b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd2d720c99f84f098a1054646aedd2a9"
I20260925 16:35:52.562240 16026 leader_election.cc:302] T 57c3d9c330f64fdeab1886abf778dabc P a114478b9c5b4e14a0de2caf167fe07b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95b51fa015394f2e892650b4f9e1bfd2, a114478b9c5b4e14a0de2caf167fe07b; no voters:
I20260925 16:35:52.562323 15959 raft_consensus.cc:3037] T 57c3d9c330f64fdeab1886abf778dabc P fd2d720c99f84f098a1054646aedd2a9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:35:52.562398 16296 raft_consensus.cc:2781] T 57c3d9c330f64fdeab1886abf778dabc P a114478b9c5b4e14a0de2caf167fe07b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:35:52.562639 16296 raft_consensus.cc:686] T 57c3d9c330f64fdeab1886abf778dabc P a114478b9c5b4e14a0de2caf167fe07b [term 1 LEADER]: Becoming Leader. State: Replica: a114478b9c5b4e14a0de2caf167fe07b, State: Running, Role: LEADER
I20260925 16:35:52.563028 16296 consensus_queue.cc:237] T 57c3d9c330f64fdeab1886abf778dabc P a114478b9c5b4e14a0de2caf167fe07b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95b51fa015394f2e892650b4f9e1bfd2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } } peers { permanent_uuid: "fd2d720c99f84f098a1054646aedd2a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37719 } } peers { permanent_uuid: "a114478b9c5b4e14a0de2caf167fe07b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43913 } }
I20260925 16:35:52.563628 15959 raft_consensus.cc:2454] T 57c3d9c330f64fdeab1886abf778dabc P fd2d720c99f84f098a1054646aedd2a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a114478b9c5b4e14a0de2caf167fe07b in term 1.
I20260925 16:35:52.566499 15813 catalog_manager.cc:5495] T 57c3d9c330f64fdeab1886abf778dabc P a114478b9c5b4e14a0de2caf167fe07b reported cstate change: term changed from 0 to 1, leader changed from <none> to a114478b9c5b4e14a0de2caf167fe07b (127.0.98.2). New cstate: current_term: 1 leader_uuid: "a114478b9c5b4e14a0de2caf167fe07b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95b51fa015394f2e892650b4f9e1bfd2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd2d720c99f84f098a1054646aedd2a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a114478b9c5b4e14a0de2caf167fe07b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43913 } health_report { overall_health: HEALTHY } } }
W20260925 16:35:52.568358 16140 tablet.cc:2369] T 57c3d9c330f64fdeab1886abf778dabc P a114478b9c5b4e14a0de2caf167fe07b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:35:52.584334 16289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:52.580707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46178 (local address 127.0.98.2:43913)
0925 16:35:52.580846 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 16:35:52.580851 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:35:52.580867 (+ 16us) server_negotiation.cc:407] Connection header received
0925 16:35:52.581039 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:35:52.581042 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:35:52.581096 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:35:52.581208 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:35:52.581443 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:52.581953 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:52.582852 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:35:52.583077 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:35:52.583215 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:35:52.583517 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:35:52.583942 (+ 425us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:35:52.584072 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 16:35:52.584112 (+ 40us) server_negotiation.cc:299] Negotiation successful
0925 16:35:52.584167 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260925 16:35:52.587757 16274 tablet.cc:2369] T 57c3d9c330f64fdeab1886abf778dabc P 95b51fa015394f2e892650b4f9e1bfd2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:35:52.589995 16227 raft_consensus.cc:1261] T 57c3d9c330f64fdeab1886abf778dabc P 95b51fa015394f2e892650b4f9e1bfd2 [term 1 FOLLOWER]: Refusing update from remote peer a114478b9c5b4e14a0de2caf167fe07b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:35:52.590070 15959 raft_consensus.cc:1261] T 57c3d9c330f64fdeab1886abf778dabc P fd2d720c99f84f098a1054646aedd2a9 [term 1 FOLLOWER]: Refusing update from remote peer a114478b9c5b4e14a0de2caf167fe07b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:35:52.590662 16296 consensus_queue.cc:1035] T 57c3d9c330f64fdeab1886abf778dabc P a114478b9c5b4e14a0de2caf167fe07b [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b51fa015394f2e892650b4f9e1bfd2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:35:52.590864 16300 consensus_queue.cc:1035] T 57c3d9c330f64fdeab1886abf778dabc P a114478b9c5b4e14a0de2caf167fe07b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd2d720c99f84f098a1054646aedd2a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:35:52.600908 16304 mvcc.cc:205] Tried to move back new op lower bound from 7333290609001975808 to 7333290608901599232. Current Snapshot: MvccSnapshot[applied={T|T < 7333290609001975808}]
I20260925 16:35:52.606923 15812 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40026:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:35:52.607077 15812 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40026:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:35:52.609454 15812 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d47abc77ad394d25b7f378cbb1ef4a60: Sending DeleteTablet for 3 replicas of tablet 57c3d9c330f64fdeab1886abf778dabc
I20260925 16:35:52.610198 15939 tablet_service.cc:1514] Processing DeleteTablet for tablet 57c3d9c330f64fdeab1886abf778dabc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:35:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47362
I20260925 16:35:52.610232 16207 tablet_service.cc:1514] Processing DeleteTablet for tablet 57c3d9c330f64fdeab1886abf778dabc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:35:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54228
I20260925 16:35:52.610713 16073 tablet_service.cc:1514] Processing DeleteTablet for tablet 57c3d9c330f64fdeab1886abf778dabc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:35:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46176
I20260925 16:35:52.610785 16313 tablet_replica.cc:330] T 57c3d9c330f64fdeab1886abf778dabc P fd2d720c99f84f098a1054646aedd2a9: stopping tablet replica
I20260925 16:35:52.611114 16313 raft_consensus.cc:2229] T 57c3d9c330f64fdeab1886abf778dabc P fd2d720c99f84f098a1054646aedd2a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:35:52.611212 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 15876
I20260925 16:35:52.611145 16314 tablet_replica.cc:330] T 57c3d9c330f64fdeab1886abf778dabc P a114478b9c5b4e14a0de2caf167fe07b: stopping tablet replica
I20260925 16:35:52.611474 16312 tablet_replica.cc:330] T 57c3d9c330f64fdeab1886abf778dabc P 95b51fa015394f2e892650b4f9e1bfd2: stopping tablet replica
I20260925 16:35:52.611625 16314 raft_consensus.cc:2229] T 57c3d9c330f64fdeab1886abf778dabc P a114478b9c5b4e14a0de2caf167fe07b [term 1 LEADER]: Raft consensus shutting down.
I20260925 16:35:52.611763 16312 raft_consensus.cc:2229] T 57c3d9c330f64fdeab1886abf778dabc P 95b51fa015394f2e892650b4f9e1bfd2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:35:52.612066 16314 raft_consensus.cc:2258] T 57c3d9c330f64fdeab1886abf778dabc P a114478b9c5b4e14a0de2caf167fe07b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:35:52.612078 16312 raft_consensus.cc:2258] T 57c3d9c330f64fdeab1886abf778dabc P 95b51fa015394f2e892650b4f9e1bfd2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:35:52.613636 16314 ts_tablet_manager.cc:1905] T 57c3d9c330f64fdeab1886abf778dabc P a114478b9c5b4e14a0de2caf167fe07b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:35:52.617197 16314 ts_tablet_manager.cc:1918] T 57c3d9c330f64fdeab1886abf778dabc P a114478b9c5b4e14a0de2caf167fe07b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:35:52.617301 16314 log.cc:1196] T 57c3d9c330f64fdeab1886abf778dabc P a114478b9c5b4e14a0de2caf167fe07b: Deleting WAL directory at /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-1/wal/wals/57c3d9c330f64fdeab1886abf778dabc
I20260925 16:35:52.617620 16314 ts_tablet_manager.cc:1939] T 57c3d9c330f64fdeab1886abf778dabc P a114478b9c5b4e14a0de2caf167fe07b: Deleting consensus metadata
I20260925 16:35:52.619159 15798 catalog_manager.cc:4841] TS a114478b9c5b4e14a0de2caf167fe07b (127.0.98.2:43913): tablet 57c3d9c330f64fdeab1886abf778dabc (table test-table [id=77692f05043f4f16a97b53f2e1d59d65]) successfully deleted
I20260925 16:35:52.620146 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 16010
W20260925 16:35:52.620150 15799 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37719 (error 108)
I20260925 16:35:52.621551 16286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:35:52.621255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37719 (local address 127.0.0.1:47378)
0925 16:35:52.621398 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 16:35:52.621470 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260925 16:35:52.621726 15799 catalog_manager.cc:4584] TS fd2d720c99f84f098a1054646aedd2a9 (127.0.98.1:37719): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 57c3d9c330f64fdeab1886abf778dabc: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37719: connect: Connection refused (error 111)
I20260925 16:35:52.622467 16312 ts_tablet_manager.cc:1905] T 57c3d9c330f64fdeab1886abf778dabc P 95b51fa015394f2e892650b4f9e1bfd2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:35:52.625054 16312 ts_tablet_manager.cc:1918] T 57c3d9c330f64fdeab1886abf778dabc P 95b51fa015394f2e892650b4f9e1bfd2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:35:52.625150 16312 log.cc:1196] T 57c3d9c330f64fdeab1886abf778dabc P 95b51fa015394f2e892650b4f9e1bfd2: Deleting WAL directory at /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-2/wal/wals/57c3d9c330f64fdeab1886abf778dabc
I20260925 16:35:52.625455 16312 ts_tablet_manager.cc:1939] T 57c3d9c330f64fdeab1886abf778dabc P 95b51fa015394f2e892650b4f9e1bfd2: Deleting consensus metadata
I20260925 16:35:52.626737 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 16144
I20260925 16:35:52.632000 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 15782
I20260925 16:36:12.638454 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39645
--webserver_interface=127.0.98.62
--webserver_port=45839
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39645
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:36:12.748952 16316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:36:12.749209 16316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:36:12.749295 16316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:36:12.753276 16316 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 16:36:12.753325 16316 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 16:36:12.753340 16316 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 16:36:12.753353 16316 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:36:12.753403 16316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:36:12.753438 16316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:36:12.753463 16316 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:36:12.753494 16316 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:36:12.758703 16316 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39645
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=45839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:36:12.759759 16316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:36:12.760810 16316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:36:12.774008 16322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:36:12.774044 16325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:36:12.774121 16316 server_base.cc:1034] running on GCE node
W20260925 16:36:12.774009 16323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:36:12.774909 16316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:36:12.776060 16316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:36:12.777232 16316 hybrid_clock.cc:648] HybridClock initialized: now 1790354172777215 us; error 32 us; skew 500 ppm
Sep 25 16:36:12 dist-test-slave-65pf krb5kdc[15769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:36:12.780078 16316 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 16:36:12.780974 16316 webserver.cc:460] Webserver started at http://127.0.98.62:45839/ using document root <none> and password file <none>
I20260925 16:36:12.781522 16316 fs_manager.cc:359] Metadata directory not provided
I20260925 16:36:12.781590 16316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:36:12.785676 16316 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260925 16:36:12.787971 16332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:12.789270 16316 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 16:36:12.789381 16316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/master-0/wal
uuid: "d47abc77ad394d25b7f378cbb1ef4a60"
format_stamp: "Formatted at 2026-09-25 16:35:51 on dist-test-slave-65pf"
server_key: "ffef4d6a0e6081cdd91a92d2015c0b1f"
server_key_iv: "0ad803c70827cf49108cee6905a210a1"
server_key_version: "encryptionkey@0"
I20260925 16:36:12.789986 16316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:36:12.801540 16316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:36:12.804616 16316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:36:12.804774 16316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:36:12.811537 16316 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39645
I20260925 16:36:12.811844 16384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39645 every 8 connection(s)
I20260925 16:36:12.812681 16316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:36:12.814862 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 16316
I20260925 16:36:12.815295 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:37719
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:39645
--webserver_port=45121
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 16:36:12.817516 16385 sys_catalog.cc:263] Verifying existing consensus state
I20260925 16:36:12.819306 16385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d47abc77ad394d25b7f378cbb1ef4a60: Bootstrap starting.
I20260925 16:36:12.836602 16385 log.cc:824] T 00000000000000000000000000000000 P d47abc77ad394d25b7f378cbb1ef4a60: Log is configured to *not* fsync() on all Append() calls
I20260925 16:36:12.845309 16385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d47abc77ad394d25b7f378cbb1ef4a60: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 16:36:12.845721 16385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d47abc77ad394d25b7f378cbb1ef4a60: Bootstrap complete.
I20260925 16:36:12.849963 16385 raft_consensus.cc:348] T 00000000000000000000000000000000 P d47abc77ad394d25b7f378cbb1ef4a60 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d47abc77ad394d25b7f378cbb1ef4a60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39645 } }
I20260925 16:36:12.850378 16385 raft_consensus.cc:729] T 00000000000000000000000000000000 P d47abc77ad394d25b7f378cbb1ef4a60 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d47abc77ad394d25b7f378cbb1ef4a60, State: Initialized, Role: FOLLOWER
I20260925 16:36:12.850997 16385 consensus_queue.cc:260] T 00000000000000000000000000000000 P d47abc77ad394d25b7f378cbb1ef4a60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d47abc77ad394d25b7f378cbb1ef4a60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39645 } }
I20260925 16:36:12.851118 16385 raft_consensus.cc:388] T 00000000000000000000000000000000 P d47abc77ad394d25b7f378cbb1ef4a60 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:36:12.851212 16385 raft_consensus.cc:482] T 00000000000000000000000000000000 P d47abc77ad394d25b7f378cbb1ef4a60 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:36:12.851329 16385 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d47abc77ad394d25b7f378cbb1ef4a60 [term 1 FOLLOWER]: Advancing to term 2
I20260925 16:36:12.853199 16385 raft_consensus.cc:504] T 00000000000000000000000000000000 P d47abc77ad394d25b7f378cbb1ef4a60 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d47abc77ad394d25b7f378cbb1ef4a60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39645 } }
I20260925 16:36:12.853534 16385 leader_election.cc:302] T 00000000000000000000000000000000 P d47abc77ad394d25b7f378cbb1ef4a60 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d47abc77ad394d25b7f378cbb1ef4a60; no voters:
I20260925 16:36:12.853868 16385 leader_election.cc:288] T 00000000000000000000000000000000 P d47abc77ad394d25b7f378cbb1ef4a60 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 16:36:12.853945 16389 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d47abc77ad394d25b7f378cbb1ef4a60 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 16:36:12.854171 16389 raft_consensus.cc:686] T 00000000000000000000000000000000 P d47abc77ad394d25b7f378cbb1ef4a60 [term 2 LEADER]: Becoming Leader. State: Replica: d47abc77ad394d25b7f378cbb1ef4a60, State: Running, Role: LEADER
I20260925 16:36:12.854468 16389 consensus_queue.cc:237] T 00000000000000000000000000000000 P d47abc77ad394d25b7f378cbb1ef4a60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d47abc77ad394d25b7f378cbb1ef4a60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39645 } }
I20260925 16:36:12.855194 16385 sys_catalog.cc:564] T 00000000000000000000000000000000 P d47abc77ad394d25b7f378cbb1ef4a60 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:36:12.855451 16391 sys_catalog.cc:455] T 00000000000000000000000000000000 P d47abc77ad394d25b7f378cbb1ef4a60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d47abc77ad394d25b7f378cbb1ef4a60. Latest consensus state: current_term: 2 leader_uuid: "d47abc77ad394d25b7f378cbb1ef4a60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d47abc77ad394d25b7f378cbb1ef4a60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39645 } } }
I20260925 16:36:12.855571 16391 sys_catalog.cc:458] T 00000000000000000000000000000000 P d47abc77ad394d25b7f378cbb1ef4a60 [sys.catalog]: This master's current role is: LEADER
I20260925 16:36:12.855792 16390 sys_catalog.cc:455] T 00000000000000000000000000000000 P d47abc77ad394d25b7f378cbb1ef4a60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d47abc77ad394d25b7f378cbb1ef4a60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d47abc77ad394d25b7f378cbb1ef4a60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39645 } } }
I20260925 16:36:12.855891 16390 sys_catalog.cc:458] T 00000000000000000000000000000000 P d47abc77ad394d25b7f378cbb1ef4a60 [sys.catalog]: This master's current role is: LEADER
I20260925 16:36:12.855933 16393 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:36:12.860707 16393 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:36:12.861326 16393 catalog_manager.cc:1259] Loaded cluster ID: b54e38f54c674aca97a7b343c71948b4
I20260925 16:36:12.861402 16393 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:36:12.863904 16393 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:36:12.864714 16393 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d47abc77ad394d25b7f378cbb1ef4a60: Loaded TSK: 0
I20260925 16:36:12.886511 16393 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d47abc77ad394d25b7f378cbb1ef4a60: Generated new TSK 1
I20260925 16:36:12.886818 16393 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 16:36:12.931450 16387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:36:12.931788 16387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:36:12.931883 16387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:36:12.935962 16387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:36:12.936065 16387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:36:12.936163 16387 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 16:36:12.941591 16387 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:37719
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=45121
--tserver_master_addrs=127.0.98.62:39645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16387
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:36:12.942711 16387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:36:12.943743 16387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:36:12.954311 16412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:36:12.954362 16413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:36:12.954542 16387 server_base.cc:1034] running on GCE node
W20260925 16:36:12.954329 16415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:36:12.955269 16387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:36:12.956080 16387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:36:12.957293 16387 hybrid_clock.cc:648] HybridClock initialized: now 1790354172957257 us; error 45 us; skew 500 ppm
Sep 25 16:36:12 dist-test-slave-65pf krb5kdc[15769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:36:12.960744 16387 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 16:36:12.961882 16387 webserver.cc:460] Webserver started at http://127.0.98.1:45121/ using document root <none> and password file <none>
I20260925 16:36:12.962626 16387 fs_manager.cc:359] Metadata directory not provided
I20260925 16:36:12.962730 16387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:36:12.967972 16387 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 16:36:12.970880 16422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:12.972394 16387 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 16:36:12.972522 16387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-0/wal
uuid: "fd2d720c99f84f098a1054646aedd2a9"
format_stamp: "Formatted at 2026-09-25 16:35:51 on dist-test-slave-65pf"
server_key: "600604f57e929abf76e71ef83692abb6"
server_key_iv: "cb60f45afaf6022379f1a9fedb1ace19"
server_key_version: "encryptionkey@0"
I20260925 16:36:12.973294 16387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:36:12.994966 16387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:36:12.998652 16387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:36:12.998845 16387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:36:12.999528 16387 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:36:13.000842 16429 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 16:36:13.005280 16387 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 16:36:13.005373 16387 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260925 16:36:13.005440 16387 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 16:36:13.007714 16387 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 16:36:13.007776 16387 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260925 16:36:13.008137 16429 tablet_bootstrap.cc:492] T 57c3d9c330f64fdeab1886abf778dabc P fd2d720c99f84f098a1054646aedd2a9: Bootstrap starting.
I20260925 16:36:13.018224 16387 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37719
I20260925 16:36:13.018421 16536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37719 every 8 connection(s)
I20260925 16:36:13.019367 16387 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
I20260925 16:36:13.022343 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 16387
I20260925 16:36:13.023036 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:43913
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:39645
--webserver_port=36163
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 16:36:13.029284 16429 log.cc:824] T 57c3d9c330f64fdeab1886abf778dabc P fd2d720c99f84f098a1054646aedd2a9: Log is configured to *not* fsync() on all Append() calls
Sep 25 16:36:13 dist-test-slave-65pf krb5kdc[15769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:36:13.037847 16540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:13.021727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42173 (local address 127.0.98.62:39645)
0925 16:36:13.024340 (+ 2613us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:13.024369 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:13.024409 (+ 40us) server_negotiation.cc:407] Connection header received
0925 16:36:13.024655 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:13.024669 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:13.025023 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:13.025343 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:36:13.027138 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:13.028088 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:13.029266 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:13.029817 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:13.030573 (+ 756us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:13.032760 (+ 2187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:36:13.032765 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:36:13.032778 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:36:13.032807 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:36:13.034567 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:36:13.035159 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:36:13.035170 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:36:13.035173 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:36:13.035286 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:36:13.035667 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:36:13.035670 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:36:13.035672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:36:13.036010 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:36:13.036149 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:13.036723 (+ 574us) server_negotiation.cc:299] Negotiation successful
0925 16:36:13.036993 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2464,"thread_start_us":135,"threads_started":1}
I20260925 16:36:13.037873 16539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:13.021652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39645 (local address 127.0.98.1:42173)
0925 16:36:13.022884 (+ 1232us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:13.022941 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 16:36:13.023889 (+ 948us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:36:13.025821 (+ 1932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:36:13.025829 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:36:13.026310 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:36:13.026950 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:13.026967 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:13.028242 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:13.028247 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:13.029120 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:13.029130 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:13.030014 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:13.030018 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:13.030220 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:13.030959 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:36:13.030999 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:36:13.032550 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:36:13.034751 (+ 2201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:36:13.034757 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:36:13.034770 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:36:13.035032 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:36:13.035410 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:36:13.035414 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:36:13.035416 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:36:13.035562 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:36:13.036137 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:36:13.036143 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:36:13.036377 (+ 234us) client_negotiation.cc:769] Sending connection context
0925 16:36:13.036575 (+ 198us) client_negotiation.cc:240] Negotiation successful
0925 16:36:13.036942 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1012,"thread_start_us":152,"threads_started":1}
I20260925 16:36:13.039919 16429 tablet_bootstrap.cc:492] T 57c3d9c330f64fdeab1886abf778dabc P fd2d720c99f84f098a1054646aedd2a9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 16:36:13.040413 16429 tablet_bootstrap.cc:492] T 57c3d9c330f64fdeab1886abf778dabc P fd2d720c99f84f098a1054646aedd2a9: Bootstrap complete.
I20260925 16:36:13.040988 16537 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39645
I20260925 16:36:13.041289 16537 heartbeater.cc:464] Registering TS with master...
I20260925 16:36:13.041774 16537 heartbeater.cc:511] Master 127.0.98.62:39645 requested a full tablet report, sending...
I20260925 16:36:13.041857 16429 ts_tablet_manager.cc:1397] T 57c3d9c330f64fdeab1886abf778dabc P fd2d720c99f84f098a1054646aedd2a9: Time spent bootstrapping tablet: real 0.034s user 0.025s sys 0.004s
I20260925 16:36:13.044528 16349 ts_manager.cc:194] Registered new tserver with Master: fd2d720c99f84f098a1054646aedd2a9 (127.0.98.1:37719)
I20260925 16:36:13.044903 16429 raft_consensus.cc:348] T 57c3d9c330f64fdeab1886abf778dabc P fd2d720c99f84f098a1054646aedd2a9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95b51fa015394f2e892650b4f9e1bfd2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } } peers { permanent_uuid: "fd2d720c99f84f098a1054646aedd2a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37719 } } peers { permanent_uuid: "a114478b9c5b4e14a0de2caf167fe07b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43913 } }
I20260925 16:36:13.045445 16429 raft_consensus.cc:729] T 57c3d9c330f64fdeab1886abf778dabc P fd2d720c99f84f098a1054646aedd2a9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd2d720c99f84f098a1054646aedd2a9, State: Initialized, Role: FOLLOWER
I20260925 16:36:13.045955 16429 consensus_queue.cc:260] T 57c3d9c330f64fdeab1886abf778dabc P fd2d720c99f84f098a1054646aedd2a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95b51fa015394f2e892650b4f9e1bfd2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } } peers { permanent_uuid: "fd2d720c99f84f098a1054646aedd2a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37719 } } peers { permanent_uuid: "a114478b9c5b4e14a0de2caf167fe07b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43913 } }
I20260925 16:36:13.047019 16429 ts_tablet_manager.cc:1428] T 57c3d9c330f64fdeab1886abf778dabc P fd2d720c99f84f098a1054646aedd2a9: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
I20260925 16:36:13.048715 16349 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42173
I20260925 16:36:13.051360 16537 heartbeater.cc:503] Master 127.0.98.62:39645 was elected leader, sending a full tablet report...
Sep 25 16:36:13 dist-test-slave-65pf krb5kdc[15769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 16:36:13.059149 16545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:13.048401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34856 (local address 127.0.98.1:37719)
0925 16:36:13.048677 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:13.048688 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:13.048711 (+ 23us) server_negotiation.cc:407] Connection header received
0925 16:36:13.048779 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:13.048784 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:13.048987 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:13.049127 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:36:13.049914 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:13.050987 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:13.052363 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:13.052902 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:13.053004 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:13.055303 (+ 2299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:36:13.055312 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:36:13.055325 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:36:13.055350 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:36:13.057178 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:36:13.057747 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:36:13.057751 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:36:13.057752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:36:13.057826 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:36:13.058147 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:36:13.058149 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:36:13.058150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:36:13.058375 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:36:13.058471 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:13.058876 (+ 405us) server_negotiation.cc:299] Negotiation successful
0925 16:36:13.059005 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":113,"threads_started":1}
I20260925 16:36:13.059203 16544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:13.047588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37719 (local address 127.0.0.1:34856)
0925 16:36:13.048013 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:13.048065 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 16:36:13.048204 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:36:13.049150 (+ 946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:36:13.049157 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:36:13.049610 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:36:13.049778 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:13.049788 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:13.051130 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:13.051134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:13.052218 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:13.052232 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:13.052993 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:13.053010 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:13.053123 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:13.053717 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:36:13.053730 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:36:13.055169 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:36:13.057292 (+ 2123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:36:13.057302 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:36:13.057308 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:36:13.057634 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:36:13.057931 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:36:13.057934 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:36:13.057936 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:36:13.058044 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:36:13.058485 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:36:13.058491 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:36:13.058645 (+ 154us) client_negotiation.cc:769] Sending connection context
0925 16:36:13.058810 (+ 165us) client_negotiation.cc:240] Negotiation successful
0925 16:36:13.059009 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":106,"threads_started":1}
I20260925 16:36:13.060207 16471 tablet_service.cc:1514] Processing DeleteTablet for tablet 57c3d9c330f64fdeab1886abf778dabc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:35:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34856
I20260925 16:36:13.061249 16546 tablet_replica.cc:330] stopping tablet replica
I20260925 16:36:13.061553 16546 raft_consensus.cc:2229] T 57c3d9c330f64fdeab1886abf778dabc P fd2d720c99f84f098a1054646aedd2a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:36:13.061793 16546 raft_consensus.cc:2258] T 57c3d9c330f64fdeab1886abf778dabc P fd2d720c99f84f098a1054646aedd2a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:36:13.062783 16546 ts_tablet_manager.cc:1905] T 57c3d9c330f64fdeab1886abf778dabc P fd2d720c99f84f098a1054646aedd2a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:36:13.066468 16546 ts_tablet_manager.cc:1918] T 57c3d9c330f64fdeab1886abf778dabc P fd2d720c99f84f098a1054646aedd2a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:36:13.066581 16546 log.cc:1196] T 57c3d9c330f64fdeab1886abf778dabc P fd2d720c99f84f098a1054646aedd2a9: Deleting WAL directory at /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-0/wal/wals/57c3d9c330f64fdeab1886abf778dabc
I20260925 16:36:13.066870 16546 ts_tablet_manager.cc:1939] T 57c3d9c330f64fdeab1886abf778dabc P fd2d720c99f84f098a1054646aedd2a9: Deleting consensus metadata
I20260925 16:36:13.068557 16334 catalog_manager.cc:4841] TS fd2d720c99f84f098a1054646aedd2a9 (127.0.98.1:37719): tablet 57c3d9c330f64fdeab1886abf778dabc (table test-table [id=77692f05043f4f16a97b53f2e1d59d65]) successfully deleted
W20260925 16:36:13.149998 16541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:36:13.150259 16541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:36:13.150348 16541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:36:13.154469 16541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:36:13.154531 16541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:36:13.154682 16541 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 16:36:13.160337 16541 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:43913
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36163
--tserver_master_addrs=127.0.98.62:39645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:36:13.161479 16541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:36:13.162652 16541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:36:13.172678 16552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:36:13.172726 16554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:36:13.172734 16551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:36:13.172873 16541 server_base.cc:1034] running on GCE node
I20260925 16:36:13.173410 16541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:36:13.174083 16541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:36:13.175254 16541 hybrid_clock.cc:648] HybridClock initialized: now 1790354173175234 us; error 27 us; skew 500 ppm
Sep 25 16:36:13 dist-test-slave-65pf krb5kdc[15769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354173, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:36:13.178131 16541 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 16:36:13.179184 16541 webserver.cc:460] Webserver started at http://127.0.98.2:36163/ using document root <none> and password file <none>
I20260925 16:36:13.179828 16541 fs_manager.cc:359] Metadata directory not provided
I20260925 16:36:13.179934 16541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:36:13.183905 16541 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 16:36:13.185930 16561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:13.187188 16541 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:36:13.187333 16541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-1/wal
uuid: "a114478b9c5b4e14a0de2caf167fe07b"
format_stamp: "Formatted at 2026-09-25 16:35:51 on dist-test-slave-65pf"
server_key: "fdb9f4d9226fa4df152bf958bbdfe358"
server_key_iv: "daf8c7d45243c5ed6221e8d143baadf0"
server_key_version: "encryptionkey@0"
I20260925 16:36:13.187886 16541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:36:13.206660 16541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:36:13.210170 16541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:36:13.210341 16541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:36:13.211045 16541 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:36:13.211885 16541 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:36:13.211935 16541 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:13.211978 16541 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:36:13.212003 16541 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:13.221624 16541 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43913
I20260925 16:36:13.221649 16674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43913 every 8 connection(s)
I20260925 16:36:13.222755 16541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
I20260925 16:36:13.224547 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 16541
I20260925 16:36:13.225081 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:37195
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:39645
--webserver_port=33447
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 16:36:13 dist-test-slave-65pf krb5kdc[15769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354173, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:36:13.238817 16540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:13.224674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39207 (local address 127.0.98.62:39645)
0925 16:36:13.224877 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:13.224884 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:13.225792 (+ 908us) server_negotiation.cc:407] Connection header received
0925 16:36:13.227168 (+ 1376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:13.227172 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:13.227230 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:13.227338 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:36:13.228520 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:13.229285 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:13.231405 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:13.231628 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:13.231710 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:13.235004 (+ 3294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:36:13.235007 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:36:13.235009 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:36:13.235038 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:36:13.236531 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:36:13.237153 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:36:13.237156 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:36:13.237157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:36:13.237228 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:36:13.237597 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:36:13.237600 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:36:13.237601 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:36:13.237767 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:36:13.237837 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:13.238462 (+ 625us) server_negotiation.cc:299] Negotiation successful
0925 16:36:13.238571 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 16:36:13.239861 16677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:13.225063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39645 (local address 127.0.98.2:39207)
0925 16:36:13.225614 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:13.225668 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 16:36:13.226943 (+ 1275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:36:13.227524 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:36:13.227532 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:36:13.227890 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:36:13.228351 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:13.228374 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:13.229488 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:13.229496 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:13.231223 (+ 1727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:13.231253 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:13.231797 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:13.231805 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:13.232056 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:13.232790 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:36:13.232820 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:36:13.234803 (+ 1983us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:36:13.236660 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:36:13.236667 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:36:13.236684 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:36:13.237042 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:36:13.237314 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:36:13.237318 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:36:13.237320 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:36:13.237500 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:36:13.237864 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:36:13.237873 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:36:13.238191 (+ 318us) client_negotiation.cc:769] Sending connection context
0925 16:36:13.238414 (+ 223us) client_negotiation.cc:240] Negotiation successful
0925 16:36:13.238813 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":184,"threads_started":1}
I20260925 16:36:13.241107 16675 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39645
I20260925 16:36:13.241364 16675 heartbeater.cc:464] Registering TS with master...
I20260925 16:36:13.241864 16675 heartbeater.cc:511] Master 127.0.98.62:39645 requested a full tablet report, sending...
I20260925 16:36:13.243005 16349 ts_manager.cc:194] Registered new tserver with Master: a114478b9c5b4e14a0de2caf167fe07b (127.0.98.2:43913)
I20260925 16:36:13.243817 16349 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39207
W20260925 16:36:13.342909 16678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:36:13.343179 16678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:36:13.343227 16678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:36:13.347417 16678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:36:13.347486 16678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:36:13.347549 16678 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 16:36:13.353010 16678 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:37195
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=33447
--tserver_master_addrs=127.0.98.62:39645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16678
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:36:13.354050 16678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:36:13.355098 16678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:36:13.365114 16684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:36:13.365114 16686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:36:13.365124 16683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:36:13.365353 16678 server_base.cc:1034] running on GCE node
I20260925 16:36:13.365821 16678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:36:13.366531 16678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:36:13.367777 16678 hybrid_clock.cc:648] HybridClock initialized: now 1790354173367751 us; error 42 us; skew 500 ppm
Sep 25 16:36:13 dist-test-slave-65pf krb5kdc[15769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354173, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:36:13.370658 16678 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 16:36:13.371609 16678 webserver.cc:460] Webserver started at http://127.0.98.3:33447/ using document root <none> and password file <none>
I20260925 16:36:13.372219 16678 fs_manager.cc:359] Metadata directory not provided
I20260925 16:36:13.372339 16678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:36:13.376273 16678 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 16:36:13.378330 16693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:13.379737 16678 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 16:36:13.379894 16678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-2/wal
uuid: "95b51fa015394f2e892650b4f9e1bfd2"
format_stamp: "Formatted at 2026-09-25 16:35:52 on dist-test-slave-65pf"
server_key: "1d2c2f1d0f4729f800dcafd2508a7234"
server_key_iv: "072658af143c19ca82e2fda2a4849fd3"
server_key_version: "encryptionkey@0"
I20260925 16:36:13.380460 16678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:36:13.392735 16678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:36:13.395916 16678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:36:13.396135 16678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:36:13.396749 16678 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:36:13.397603 16678 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:36:13.397678 16678 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:13.397751 16678 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:36:13.397791 16678 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:13.407630 16678 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37195
I20260925 16:36:13.407650 16806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37195 every 8 connection(s)
I20260925 16:36:13.408694 16678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
I20260925 16:36:13.411554 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 16678
Sep 25 16:36:13 dist-test-slave-65pf krb5kdc[15769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354173, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:36:13.421025 16811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:13.412260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36592 (local address 127.0.98.62:39645)
0925 16:36:13.412526 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:13.412531 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:13.412659 (+ 128us) server_negotiation.cc:407] Connection header received
0925 16:36:13.412778 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:13.412781 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:13.412831 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:13.412931 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:36:13.413736 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:13.414307 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:13.415302 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:13.415537 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:13.415659 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:13.417177 (+ 1518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:36:13.417189 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:36:13.417192 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:36:13.417216 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:36:13.419007 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:36:13.419469 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:36:13.419475 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:36:13.419477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:36:13.419603 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:36:13.419963 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:36:13.419966 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:36:13.419967 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:36:13.420231 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:36:13.420424 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:13.420669 (+ 245us) server_negotiation.cc:299] Negotiation successful
0925 16:36:13.420799 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":110,"threads_started":1}
I20260925 16:36:13.423285 16540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:13.410572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55713 (local address 127.0.98.62:39645)
0925 16:36:13.410748 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:13.410752 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:13.412334 (+ 1582us) server_negotiation.cc:407] Connection header received
0925 16:36:13.413480 (+ 1146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:13.413483 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:13.413528 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:13.413601 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:36:13.414759 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:13.415245 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:13.416349 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:13.416568 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:13.416675 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:13.419461 (+ 2786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:36:13.419464 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:36:13.419466 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:36:13.419487 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:36:13.421184 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:36:13.421742 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:36:13.421745 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:36:13.421747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:36:13.421824 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:36:13.422164 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:36:13.422166 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:36:13.422167 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:36:13.422325 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:36:13.422398 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:13.423027 (+ 629us) server_negotiation.cc:299] Negotiation successful
0925 16:36:13.423135 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50,"spinlock_wait_cycles":1792}
I20260925 16:36:13.424131 16809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:13.410998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39645 (local address 127.0.98.3:55713)
0925 16:36:13.412177 (+ 1179us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:13.412235 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 16:36:13.413282 (+ 1047us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:36:13.413784 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:36:13.413791 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:36:13.414133 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:36:13.414614 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:13.414627 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:13.415365 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:13.415371 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:13.416221 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:13.416230 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:13.416710 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:13.416714 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:13.416885 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:13.417438 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:36:13.417479 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:36:13.419289 (+ 1810us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:36:13.421298 (+ 2009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:36:13.421304 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:36:13.421319 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:36:13.421635 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:36:13.421915 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:36:13.421918 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:36:13.421920 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:36:13.422065 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:36:13.422421 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:36:13.422427 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:36:13.422751 (+ 324us) client_negotiation.cc:769] Sending connection context
0925 16:36:13.422974 (+ 223us) client_negotiation.cc:240] Negotiation successful
0925 16:36:13.423302 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":952,"thread_start_us":565,"threads_started":1}
I20260925 16:36:13.425100 16807 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39645
I20260925 16:36:13.425318 16807 heartbeater.cc:464] Registering TS with master...
I20260925 16:36:13.425763 16807 heartbeater.cc:511] Master 127.0.98.62:39645 requested a full tablet report, sending...
I20260925 16:36:13.426734 16349 ts_manager.cc:194] Registered new tserver with Master: 95b51fa015394f2e892650b4f9e1bfd2 (127.0.98.3:37195)
I20260925 16:36:13.427244 16349 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55713
I20260925 16:36:13.434666 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 16:36:13.438334 16540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:13.435290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36600 (local address 127.0.98.62:39645)
0925 16:36:13.435414 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:13.435417 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:13.435581 (+ 164us) server_negotiation.cc:407] Connection header received
0925 16:36:13.435701 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:13.435703 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:13.435798 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:13.435883 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:36:13.436088 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:13.436577 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:13.437377 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:13.437575 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:13.437647 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:13.437914 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:36:13.438049 (+ 135us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0925 16:36:13.438240 (+ 191us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36600: token expired
Metrics: {"server-negotiator.queue_time_us":42}
W20260925 16:36:13.438437 16540 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36600: token expired
W20260925 16:36:13.438791 16278 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:39645)
I20260925 16:36:13.445688 16540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:13.439217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36616 (local address 127.0.98.62:39645)
0925 16:36:13.439321 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:13.439324 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:13.439335 (+ 11us) server_negotiation.cc:407] Connection header received
0925 16:36:13.439363 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:13.439365 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:13.439398 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:13.439467 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:36:13.440037 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:13.440505 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:13.441320 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:13.441530 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:13.441613 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:13.442990 (+ 1377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:36:13.442993 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:36:13.442995 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:36:13.443016 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:36:13.444307 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:36:13.444693 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:36:13.444695 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:36:13.444697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:36:13.444765 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:36:13.445006 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:36:13.445008 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:36:13.445009 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:36:13.445171 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:36:13.445269 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:13.445464 (+ 195us) server_negotiation.cc:299] Negotiation successful
0925 16:36:13.445570 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 16:36:13.448360 16349 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36616:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 16:36:13.451810 16349 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:36:13.459729 16471 tablet_service.cc:1467] Processing CreateTablet for tablet 2e8df4ca01f74f568b40105f5258ae87 (DEFAULT_TABLE table=test-table [id=0c59683334cd44e0b8dc279337709f25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:36:13.460448 16471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e8df4ca01f74f568b40105f5258ae87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:36:13.463356 16429 tablet_bootstrap.cc:492] T 2e8df4ca01f74f568b40105f5258ae87 P fd2d720c99f84f098a1054646aedd2a9: Bootstrap starting.
I20260925 16:36:13.464294 16429 tablet_bootstrap.cc:654] T 2e8df4ca01f74f568b40105f5258ae87 P fd2d720c99f84f098a1054646aedd2a9: Neither blocks nor log segments found. Creating new log.
I20260925 16:36:13.464726 16816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:13.458880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43913 (local address 127.0.0.1:56482)
0925 16:36:13.459166 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:13.459181 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:36:13.459257 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:36:13.460054 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:36:13.460058 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:36:13.460084 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:36:13.460217 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:13.460223 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:13.461625 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:13.461631 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:13.462776 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:13.462787 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:13.464120 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:13.464123 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:13.464250 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:13.464320 (+ 70us) client_negotiation.cc:769] Sending connection context
0925 16:36:13.464428 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 16:36:13.464531 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":138,"threads_started":1}
I20260925 16:36:13.465240 16544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:13.458638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37195 (local address 127.0.0.1:37040)
0925 16:36:13.458757 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:13.458773 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:36:13.458854 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:36:13.460189 (+ 1335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:36:13.460192 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:36:13.460204 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:36:13.460275 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:13.460281 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:13.462123 (+ 1842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:13.462127 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:13.463238 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:13.463246 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:13.464382 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:13.464389 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:13.464500 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:13.464523 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 16:36:13.464575 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 16:36:13.464664 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 16:36:13.465245 16817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:13.459078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56482 (local address 127.0.98.2:43913)
0925 16:36:13.459545 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:13.459562 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:13.459587 (+ 25us) server_negotiation.cc:407] Connection header received
0925 16:36:13.459678 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:13.459683 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:13.459866 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:13.460413 (+ 547us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:36:13.460606 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:13.461507 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:13.462930 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:13.463986 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:13.464107 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:13.464270 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:13.464964 (+ 694us) server_negotiation.cc:299] Negotiation successful
0925 16:36:13.465097 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":134,"threads_started":1}
I20260925 16:36:13.466120 16818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:13.459329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37040 (local address 127.0.98.3:37195)
0925 16:36:13.459708 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:13.459720 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:13.459745 (+ 25us) server_negotiation.cc:407] Connection header received
0925 16:36:13.459824 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:13.459829 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:13.460043 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:13.460214 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:36:13.460718 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:13.461986 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:13.463382 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:13.464198 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:13.464827 (+ 629us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:13.465043 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:13.465096 (+ 53us) server_negotiation.cc:299] Negotiation successful
0925 16:36:13.465187 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":164,"threads_started":1}
I20260925 16:36:13.466260 16429 tablet_bootstrap.cc:492] T 2e8df4ca01f74f568b40105f5258ae87 P fd2d720c99f84f098a1054646aedd2a9: No bootstrap required, opened a new log
I20260925 16:36:13.466347 16429 ts_tablet_manager.cc:1397] T 2e8df4ca01f74f568b40105f5258ae87 P fd2d720c99f84f098a1054646aedd2a9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 16:36:13.466837 16429 raft_consensus.cc:348] T 2e8df4ca01f74f568b40105f5258ae87 P fd2d720c99f84f098a1054646aedd2a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95b51fa015394f2e892650b4f9e1bfd2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } } peers { permanent_uuid: "a114478b9c5b4e14a0de2caf167fe07b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43913 } } peers { permanent_uuid: "fd2d720c99f84f098a1054646aedd2a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37719 } }
I20260925 16:36:13.466970 16429 raft_consensus.cc:374] T 2e8df4ca01f74f568b40105f5258ae87 P fd2d720c99f84f098a1054646aedd2a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:36:13.467006 16429 raft_consensus.cc:729] T 2e8df4ca01f74f568b40105f5258ae87 P fd2d720c99f84f098a1054646aedd2a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd2d720c99f84f098a1054646aedd2a9, State: Initialized, Role: FOLLOWER
I20260925 16:36:13.467126 16429 consensus_queue.cc:260] T 2e8df4ca01f74f568b40105f5258ae87 P fd2d720c99f84f098a1054646aedd2a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95b51fa015394f2e892650b4f9e1bfd2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } } peers { permanent_uuid: "a114478b9c5b4e14a0de2caf167fe07b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43913 } } peers { permanent_uuid: "fd2d720c99f84f098a1054646aedd2a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37719 } }
I20260925 16:36:13.467365 16429 ts_tablet_manager.cc:1428] T 2e8df4ca01f74f568b40105f5258ae87 P fd2d720c99f84f098a1054646aedd2a9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 16:36:13.468369 16609 tablet_service.cc:1467] Processing CreateTablet for tablet 2e8df4ca01f74f568b40105f5258ae87 (DEFAULT_TABLE table=test-table [id=0c59683334cd44e0b8dc279337709f25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:36:13.468700 16741 tablet_service.cc:1467] Processing CreateTablet for tablet 2e8df4ca01f74f568b40105f5258ae87 (DEFAULT_TABLE table=test-table [id=0c59683334cd44e0b8dc279337709f25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:36:13.469317 16609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e8df4ca01f74f568b40105f5258ae87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:36:13.469491 16741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e8df4ca01f74f568b40105f5258ae87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:36:13.476843 16820 tablet_bootstrap.cc:492] T 2e8df4ca01f74f568b40105f5258ae87 P a114478b9c5b4e14a0de2caf167fe07b: Bootstrap starting.
I20260925 16:36:13.477319 16821 tablet_bootstrap.cc:492] T 2e8df4ca01f74f568b40105f5258ae87 P 95b51fa015394f2e892650b4f9e1bfd2: Bootstrap starting.
I20260925 16:36:13.479254 16821 tablet_bootstrap.cc:654] T 2e8df4ca01f74f568b40105f5258ae87 P 95b51fa015394f2e892650b4f9e1bfd2: Neither blocks nor log segments found. Creating new log.
I20260925 16:36:13.479641 16820 tablet_bootstrap.cc:654] T 2e8df4ca01f74f568b40105f5258ae87 P a114478b9c5b4e14a0de2caf167fe07b: Neither blocks nor log segments found. Creating new log.
I20260925 16:36:13.480384 16821 log.cc:824] T 2e8df4ca01f74f568b40105f5258ae87 P 95b51fa015394f2e892650b4f9e1bfd2: Log is configured to *not* fsync() on all Append() calls
I20260925 16:36:13.480391 16820 log.cc:824] T 2e8df4ca01f74f568b40105f5258ae87 P a114478b9c5b4e14a0de2caf167fe07b: Log is configured to *not* fsync() on all Append() calls
I20260925 16:36:13.482445 16820 tablet_bootstrap.cc:492] T 2e8df4ca01f74f568b40105f5258ae87 P a114478b9c5b4e14a0de2caf167fe07b: No bootstrap required, opened a new log
I20260925 16:36:13.482445 16821 tablet_bootstrap.cc:492] T 2e8df4ca01f74f568b40105f5258ae87 P 95b51fa015394f2e892650b4f9e1bfd2: No bootstrap required, opened a new log
I20260925 16:36:13.482692 16820 ts_tablet_manager.cc:1397] T 2e8df4ca01f74f568b40105f5258ae87 P a114478b9c5b4e14a0de2caf167fe07b: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 16:36:13.482698 16821 ts_tablet_manager.cc:1397] T 2e8df4ca01f74f568b40105f5258ae87 P 95b51fa015394f2e892650b4f9e1bfd2: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 16:36:13.485805 16820 raft_consensus.cc:348] T 2e8df4ca01f74f568b40105f5258ae87 P a114478b9c5b4e14a0de2caf167fe07b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95b51fa015394f2e892650b4f9e1bfd2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } } peers { permanent_uuid: "a114478b9c5b4e14a0de2caf167fe07b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43913 } } peers { permanent_uuid: "fd2d720c99f84f098a1054646aedd2a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37719 } }
I20260925 16:36:13.485806 16821 raft_consensus.cc:348] T 2e8df4ca01f74f568b40105f5258ae87 P 95b51fa015394f2e892650b4f9e1bfd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95b51fa015394f2e892650b4f9e1bfd2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } } peers { permanent_uuid: "a114478b9c5b4e14a0de2caf167fe07b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43913 } } peers { permanent_uuid: "fd2d720c99f84f098a1054646aedd2a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37719 } }
I20260925 16:36:13.486052 16820 raft_consensus.cc:374] T 2e8df4ca01f74f568b40105f5258ae87 P a114478b9c5b4e14a0de2caf167fe07b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:36:13.486075 16821 raft_consensus.cc:374] T 2e8df4ca01f74f568b40105f5258ae87 P 95b51fa015394f2e892650b4f9e1bfd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:36:13.486111 16820 raft_consensus.cc:729] T 2e8df4ca01f74f568b40105f5258ae87 P a114478b9c5b4e14a0de2caf167fe07b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a114478b9c5b4e14a0de2caf167fe07b, State: Initialized, Role: FOLLOWER
I20260925 16:36:13.486138 16821 raft_consensus.cc:729] T 2e8df4ca01f74f568b40105f5258ae87 P 95b51fa015394f2e892650b4f9e1bfd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95b51fa015394f2e892650b4f9e1bfd2, State: Initialized, Role: FOLLOWER
I20260925 16:36:13.486610 16820 consensus_queue.cc:260] T 2e8df4ca01f74f568b40105f5258ae87 P a114478b9c5b4e14a0de2caf167fe07b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95b51fa015394f2e892650b4f9e1bfd2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } } peers { permanent_uuid: "a114478b9c5b4e14a0de2caf167fe07b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43913 } } peers { permanent_uuid: "fd2d720c99f84f098a1054646aedd2a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37719 } }
I20260925 16:36:13.486610 16821 consensus_queue.cc:260] T 2e8df4ca01f74f568b40105f5258ae87 P 95b51fa015394f2e892650b4f9e1bfd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95b51fa015394f2e892650b4f9e1bfd2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } } peers { permanent_uuid: "a114478b9c5b4e14a0de2caf167fe07b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43913 } } peers { permanent_uuid: "fd2d720c99f84f098a1054646aedd2a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37719 } }
I20260925 16:36:13.487325 16675 heartbeater.cc:503] Master 127.0.98.62:39645 was elected leader, sending a full tablet report...
I20260925 16:36:13.487493 16807 heartbeater.cc:503] Master 127.0.98.62:39645 was elected leader, sending a full tablet report...
I20260925 16:36:13.487650 16821 ts_tablet_manager.cc:1428] T 2e8df4ca01f74f568b40105f5258ae87 P 95b51fa015394f2e892650b4f9e1bfd2: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260925 16:36:13.488445 16820 ts_tablet_manager.cc:1428] T 2e8df4ca01f74f568b40105f5258ae87 P a114478b9c5b4e14a0de2caf167fe07b: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260925 16:36:13.521011 16538 tablet.cc:2369] T 2e8df4ca01f74f568b40105f5258ae87 P fd2d720c99f84f098a1054646aedd2a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:36:13.571050 16824 raft_consensus.cc:482] T 2e8df4ca01f74f568b40105f5258ae87 P a114478b9c5b4e14a0de2caf167fe07b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:36:13.571270 16824 raft_consensus.cc:504] T 2e8df4ca01f74f568b40105f5258ae87 P a114478b9c5b4e14a0de2caf167fe07b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95b51fa015394f2e892650b4f9e1bfd2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } } peers { permanent_uuid: "a114478b9c5b4e14a0de2caf167fe07b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43913 } } peers { permanent_uuid: "fd2d720c99f84f098a1054646aedd2a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37719 } }
I20260925 16:36:13.572497 16824 leader_election.cc:288] T 2e8df4ca01f74f568b40105f5258ae87 P a114478b9c5b4e14a0de2caf167fe07b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 95b51fa015394f2e892650b4f9e1bfd2 (127.0.98.3:37195), fd2d720c99f84f098a1054646aedd2a9 (127.0.98.1:37719)
I20260925 16:36:13.577661 16827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:13.572868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45899 (local address 127.0.98.1:37719)
0925 16:36:13.573147 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:13.573152 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:13.573267 (+ 115us) server_negotiation.cc:407] Connection header received
0925 16:36:13.573340 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:13.573346 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:13.573417 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:13.573545 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:36:13.573803 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:13.574532 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:13.576272 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:13.576935 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:13.577030 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:13.577185 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:13.577450 (+ 265us) server_negotiation.cc:299] Negotiation successful
0925 16:36:13.577524 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":85,"threads_started":1}
I20260925 16:36:13.577659 16818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:13.572828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37191 (local address 127.0.98.3:37195)
0925 16:36:13.572984 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:13.572991 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:13.573010 (+ 19us) server_negotiation.cc:407] Connection header received
0925 16:36:13.573219 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:13.573222 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:13.573298 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:13.573399 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:36:13.573682 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:13.574473 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:13.576272 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:13.576879 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:13.577002 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:13.577072 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:13.577450 (+ 378us) server_negotiation.cc:299] Negotiation successful
0925 16:36:13.577522 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260925 16:36:13.577669 16677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:13.572693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37195 (local address 127.0.98.2:37191)
0925 16:36:13.572948 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:13.572964 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:36:13.573066 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:36:13.573418 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:36:13.573420 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:36:13.573433 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:36:13.573567 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:13.573574 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:13.574621 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:13.574628 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:13.576146 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:13.576157 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:13.577056 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:13.577060 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:13.577244 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:13.577344 (+ 100us) client_negotiation.cc:769] Sending connection context
0925 16:36:13.577420 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 16:36:13.577519 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":219}
I20260925 16:36:13.577682 16826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:13.572794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37719 (local address 127.0.98.2:45899)
0925 16:36:13.573166 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:13.573178 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 16:36:13.573270 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:36:13.573564 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:36:13.573566 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:36:13.573580 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:36:13.573695 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:13.573700 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:13.574679 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:13.574682 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:13.576146 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:13.576157 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:13.577056 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:13.577060 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:13.577288 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:13.577342 (+ 54us) client_negotiation.cc:769] Sending connection context
0925 16:36:13.577420 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 16:36:13.577519 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":199,"thread_start_us":109,"threads_started":1}
I20260925 16:36:13.578400 16761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e8df4ca01f74f568b40105f5258ae87" candidate_uuid: "a114478b9c5b4e14a0de2caf167fe07b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95b51fa015394f2e892650b4f9e1bfd2" is_pre_election: true
I20260925 16:36:13.578389 16491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e8df4ca01f74f568b40105f5258ae87" candidate_uuid: "a114478b9c5b4e14a0de2caf167fe07b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd2d720c99f84f098a1054646aedd2a9" is_pre_election: true
I20260925 16:36:13.578740 16761 raft_consensus.cc:2454] T 2e8df4ca01f74f568b40105f5258ae87 P 95b51fa015394f2e892650b4f9e1bfd2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a114478b9c5b4e14a0de2caf167fe07b in term 0.
I20260925 16:36:13.578740 16491 raft_consensus.cc:2454] T 2e8df4ca01f74f568b40105f5258ae87 P fd2d720c99f84f098a1054646aedd2a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a114478b9c5b4e14a0de2caf167fe07b in term 0.
I20260925 16:36:13.579192 16563 leader_election.cc:302] T 2e8df4ca01f74f568b40105f5258ae87 P a114478b9c5b4e14a0de2caf167fe07b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 95b51fa015394f2e892650b4f9e1bfd2, a114478b9c5b4e14a0de2caf167fe07b, fd2d720c99f84f098a1054646aedd2a9; no voters:
I20260925 16:36:13.579521 16824 raft_consensus.cc:2781] T 2e8df4ca01f74f568b40105f5258ae87 P a114478b9c5b4e14a0de2caf167fe07b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:36:13.579624 16824 raft_consensus.cc:482] T 2e8df4ca01f74f568b40105f5258ae87 P a114478b9c5b4e14a0de2caf167fe07b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:36:13.579699 16824 raft_consensus.cc:3037] T 2e8df4ca01f74f568b40105f5258ae87 P a114478b9c5b4e14a0de2caf167fe07b [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:36:13.580821 16824 raft_consensus.cc:504] T 2e8df4ca01f74f568b40105f5258ae87 P a114478b9c5b4e14a0de2caf167fe07b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95b51fa015394f2e892650b4f9e1bfd2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } } peers { permanent_uuid: "a114478b9c5b4e14a0de2caf167fe07b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43913 } } peers { permanent_uuid: "fd2d720c99f84f098a1054646aedd2a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37719 } }
I20260925 16:36:13.581147 16824 leader_election.cc:288] T 2e8df4ca01f74f568b40105f5258ae87 P a114478b9c5b4e14a0de2caf167fe07b [CANDIDATE]: Term 1 election: Requested vote from peers 95b51fa015394f2e892650b4f9e1bfd2 (127.0.98.3:37195), fd2d720c99f84f098a1054646aedd2a9 (127.0.98.1:37719)
I20260925 16:36:13.581444 16761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e8df4ca01f74f568b40105f5258ae87" candidate_uuid: "a114478b9c5b4e14a0de2caf167fe07b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95b51fa015394f2e892650b4f9e1bfd2"
I20260925 16:36:13.581573 16761 raft_consensus.cc:3037] T 2e8df4ca01f74f568b40105f5258ae87 P 95b51fa015394f2e892650b4f9e1bfd2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:36:13.581578 16491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e8df4ca01f74f568b40105f5258ae87" candidate_uuid: "a114478b9c5b4e14a0de2caf167fe07b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd2d720c99f84f098a1054646aedd2a9"
I20260925 16:36:13.581724 16491 raft_consensus.cc:3037] T 2e8df4ca01f74f568b40105f5258ae87 P fd2d720c99f84f098a1054646aedd2a9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:36:13.582679 16761 raft_consensus.cc:2454] T 2e8df4ca01f74f568b40105f5258ae87 P 95b51fa015394f2e892650b4f9e1bfd2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a114478b9c5b4e14a0de2caf167fe07b in term 1.
I20260925 16:36:13.582760 16491 raft_consensus.cc:2454] T 2e8df4ca01f74f568b40105f5258ae87 P fd2d720c99f84f098a1054646aedd2a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a114478b9c5b4e14a0de2caf167fe07b in term 1.
I20260925 16:36:13.583005 16562 leader_election.cc:302] T 2e8df4ca01f74f568b40105f5258ae87 P a114478b9c5b4e14a0de2caf167fe07b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95b51fa015394f2e892650b4f9e1bfd2, a114478b9c5b4e14a0de2caf167fe07b; no voters:
I20260925 16:36:13.583181 16824 raft_consensus.cc:2781] T 2e8df4ca01f74f568b40105f5258ae87 P a114478b9c5b4e14a0de2caf167fe07b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:36:13.583389 16824 raft_consensus.cc:686] T 2e8df4ca01f74f568b40105f5258ae87 P a114478b9c5b4e14a0de2caf167fe07b [term 1 LEADER]: Becoming Leader. State: Replica: a114478b9c5b4e14a0de2caf167fe07b, State: Running, Role: LEADER
I20260925 16:36:13.583727 16824 consensus_queue.cc:237] T 2e8df4ca01f74f568b40105f5258ae87 P a114478b9c5b4e14a0de2caf167fe07b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95b51fa015394f2e892650b4f9e1bfd2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } } peers { permanent_uuid: "a114478b9c5b4e14a0de2caf167fe07b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43913 } } peers { permanent_uuid: "fd2d720c99f84f098a1054646aedd2a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37719 } }
I20260925 16:36:13.586983 16348 catalog_manager.cc:5495] T 2e8df4ca01f74f568b40105f5258ae87 P a114478b9c5b4e14a0de2caf167fe07b reported cstate change: term changed from 0 to 1, leader changed from <none> to a114478b9c5b4e14a0de2caf167fe07b (127.0.98.2). New cstate: current_term: 1 leader_uuid: "a114478b9c5b4e14a0de2caf167fe07b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95b51fa015394f2e892650b4f9e1bfd2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a114478b9c5b4e14a0de2caf167fe07b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fd2d720c99f84f098a1054646aedd2a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37719 } health_report { overall_health: UNKNOWN } } }
I20260925 16:36:13.618556 16817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:13.615048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56486 (local address 127.0.98.2:43913)
0925 16:36:13.615193 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:13.615198 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:13.615214 (+ 16us) server_negotiation.cc:407] Connection header received
0925 16:36:13.615351 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:13.615353 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:13.615409 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:13.615526 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:36:13.615756 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:13.616264 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:13.617095 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:13.617322 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:13.617430 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:13.617726 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:36:13.618126 (+ 400us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:36:13.618260 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:13.618329 (+ 69us) server_negotiation.cc:299] Negotiation successful
0925 16:36:13.618383 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 16:36:13.624238 16491 raft_consensus.cc:1261] T 2e8df4ca01f74f568b40105f5258ae87 P fd2d720c99f84f098a1054646aedd2a9 [term 1 FOLLOWER]: Refusing update from remote peer a114478b9c5b4e14a0de2caf167fe07b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:36:13.624243 16761 raft_consensus.cc:1261] T 2e8df4ca01f74f568b40105f5258ae87 P 95b51fa015394f2e892650b4f9e1bfd2 [term 1 FOLLOWER]: Refusing update from remote peer a114478b9c5b4e14a0de2caf167fe07b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:36:13.624943 16828 consensus_queue.cc:1035] T 2e8df4ca01f74f568b40105f5258ae87 P a114478b9c5b4e14a0de2caf167fe07b [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b51fa015394f2e892650b4f9e1bfd2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:36:13.625123 16824 consensus_queue.cc:1035] T 2e8df4ca01f74f568b40105f5258ae87 P a114478b9c5b4e14a0de2caf167fe07b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd2d720c99f84f098a1054646aedd2a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:36:13.631699 16831 mvcc.cc:205] Tried to move back new op lower bound from 7333290695157805056 to 7333290695001468928. Current Snapshot: MvccSnapshot[applied={T|T < 7333290695157805056}]
I20260925 16:36:13.632230 16832 mvcc.cc:205] Tried to move back new op lower bound from 7333290695157805056 to 7333290695001468928. Current Snapshot: MvccSnapshot[applied={T|T < 7333290695157805056}]
I20260925 16:36:13.639761 16349 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:36:13.639959 16349 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:36:13.642132 16349 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d47abc77ad394d25b7f378cbb1ef4a60: Sending DeleteTablet for 3 replicas of tablet 2e8df4ca01f74f568b40105f5258ae87
I20260925 16:36:13.642868 16609 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e8df4ca01f74f568b40105f5258ae87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:36:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56482
I20260925 16:36:13.642874 16741 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e8df4ca01f74f568b40105f5258ae87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:36:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37040
I20260925 16:36:13.642910 16471 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e8df4ca01f74f568b40105f5258ae87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:36:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34856
I20260925 16:36:13.643353 16841 tablet_replica.cc:330] T 2e8df4ca01f74f568b40105f5258ae87 P a114478b9c5b4e14a0de2caf167fe07b: stopping tablet replica
I20260925 16:36:13.643487 16840 tablet_replica.cc:330] T 2e8df4ca01f74f568b40105f5258ae87 P fd2d720c99f84f098a1054646aedd2a9: stopping tablet replica
I20260925 16:36:13.643532 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 16387
I20260925 16:36:13.643669 16841 raft_consensus.cc:2229] T 2e8df4ca01f74f568b40105f5258ae87 P a114478b9c5b4e14a0de2caf167fe07b [term 1 LEADER]: Raft consensus shutting down.
I20260925 16:36:13.644153 16841 raft_consensus.cc:2258] T 2e8df4ca01f74f568b40105f5258ae87 P a114478b9c5b4e14a0de2caf167fe07b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:36:13.645478 16841 ts_tablet_manager.cc:1905] T 2e8df4ca01f74f568b40105f5258ae87 P a114478b9c5b4e14a0de2caf167fe07b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:36:13.647732 16842 tablet_replica.cc:330] T 2e8df4ca01f74f568b40105f5258ae87 P 95b51fa015394f2e892650b4f9e1bfd2: stopping tablet replica
I20260925 16:36:13.648005 16841 ts_tablet_manager.cc:1918] T 2e8df4ca01f74f568b40105f5258ae87 P a114478b9c5b4e14a0de2caf167fe07b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:36:13.648074 16842 raft_consensus.cc:2229] T 2e8df4ca01f74f568b40105f5258ae87 P 95b51fa015394f2e892650b4f9e1bfd2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:36:13.648162 16841 log.cc:1196] T 2e8df4ca01f74f568b40105f5258ae87 P a114478b9c5b4e14a0de2caf167fe07b: Deleting WAL directory at /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-1/wal/wals/2e8df4ca01f74f568b40105f5258ae87
I20260925 16:36:13.648435 16842 raft_consensus.cc:2258] T 2e8df4ca01f74f568b40105f5258ae87 P 95b51fa015394f2e892650b4f9e1bfd2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:36:13.648631 16841 ts_tablet_manager.cc:1939] T 2e8df4ca01f74f568b40105f5258ae87 P a114478b9c5b4e14a0de2caf167fe07b: Deleting consensus metadata
I20260925 16:36:13.649863 16333 catalog_manager.cc:4841] TS a114478b9c5b4e14a0de2caf167fe07b (127.0.98.2:43913): tablet 2e8df4ca01f74f568b40105f5258ae87 (table test-table [id=0c59683334cd44e0b8dc279337709f25]) successfully deleted
I20260925 16:36:13.649963 16842 ts_tablet_manager.cc:1905] T 2e8df4ca01f74f568b40105f5258ae87 P 95b51fa015394f2e892650b4f9e1bfd2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 16:36:13.651510 16334 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37719 (error 108)
I20260925 16:36:13.652323 16842 ts_tablet_manager.cc:1918] T 2e8df4ca01f74f568b40105f5258ae87 P 95b51fa015394f2e892650b4f9e1bfd2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:36:13.652374 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 16541
I20260925 16:36:13.653600 16842 log.cc:1196] T 2e8df4ca01f74f568b40105f5258ae87 P 95b51fa015394f2e892650b4f9e1bfd2: Deleting WAL directory at /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-2/wal/wals/2e8df4ca01f74f568b40105f5258ae87
I20260925 16:36:13.653987 16842 ts_tablet_manager.cc:1939] T 2e8df4ca01f74f568b40105f5258ae87 P 95b51fa015394f2e892650b4f9e1bfd2: Deleting consensus metadata
I20260925 16:36:13.654376 16544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:13.654026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37719 (local address 127.0.0.1:34866)
0925 16:36:13.654193 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:13.654276 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 16:36:13.654554 16334 catalog_manager.cc:4584] TS fd2d720c99f84f098a1054646aedd2a9 (127.0.98.1:37719): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2e8df4ca01f74f568b40105f5258ae87: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37719: connect: Connection refused (error 111)
I20260925 16:36:13.655766 16333 catalog_manager.cc:4841] TS 95b51fa015394f2e892650b4f9e1bfd2 (127.0.98.3:37195): tablet 2e8df4ca01f74f568b40105f5258ae87 (table test-table [id=0c59683334cd44e0b8dc279337709f25]) successfully deleted
I20260925 16:36:13.659216 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 16678
I20260925 16:36:13.667460 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 16316
I20260925 16:36:33.674015 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39645
--webserver_interface=127.0.98.62
--webserver_port=45839
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39645
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:36:33.784106 16844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:36:33.784399 16844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:36:33.784471 16844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:36:33.788477 16844 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 16:36:33.788549 16844 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 16:36:33.788576 16844 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 16:36:33.788605 16844 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:36:33.788645 16844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:36:33.788682 16844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:36:33.788743 16844 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:36:33.788779 16844 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:36:33.793888 16844 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39645
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=45839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:36:33.794976 16844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:36:33.795992 16844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:36:33.804455 16849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:36:33.804594 16844 server_base.cc:1034] running on GCE node
W20260925 16:36:33.804471 16850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:36:33.804793 16852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:36:33.805338 16844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:36:33.806449 16844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:36:33.807641 16844 hybrid_clock.cc:648] HybridClock initialized: now 1790354193807624 us; error 29 us; skew 500 ppm
Sep 25 16:36:33 dist-test-slave-65pf krb5kdc[15769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354193, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:36:33.810395 16844 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 16:36:33.811322 16844 webserver.cc:460] Webserver started at http://127.0.98.62:45839/ using document root <none> and password file <none>
I20260925 16:36:33.811923 16844 fs_manager.cc:359] Metadata directory not provided
I20260925 16:36:33.812014 16844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:36:33.815840 16844 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.002s
I20260925 16:36:33.818037 16859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:33.819303 16844 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 16:36:33.819443 16844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/master-0/wal
uuid: "d47abc77ad394d25b7f378cbb1ef4a60"
format_stamp: "Formatted at 2026-09-25 16:35:51 on dist-test-slave-65pf"
server_key: "ffef4d6a0e6081cdd91a92d2015c0b1f"
server_key_iv: "0ad803c70827cf49108cee6905a210a1"
server_key_version: "encryptionkey@0"
I20260925 16:36:33.820003 16844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:36:33.831903 16844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:36:33.835096 16844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:36:33.835305 16844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:36:33.841964 16844 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39645
I20260925 16:36:33.842250 16911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39645 every 8 connection(s)
I20260925 16:36:33.843111 16844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:36:33.847432 16912 sys_catalog.cc:263] Verifying existing consensus state
I20260925 16:36:33.849035 16912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d47abc77ad394d25b7f378cbb1ef4a60: Bootstrap starting.
I20260925 16:36:33.850322 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 16844
I20260925 16:36:33.850867 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:37719
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:39645
--webserver_port=45121
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 16:36:33.866695 16912 log.cc:824] T 00000000000000000000000000000000 P d47abc77ad394d25b7f378cbb1ef4a60: Log is configured to *not* fsync() on all Append() calls
I20260925 16:36:33.877544 16912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d47abc77ad394d25b7f378cbb1ef4a60: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 16:36:33.877934 16912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d47abc77ad394d25b7f378cbb1ef4a60: Bootstrap complete.
I20260925 16:36:33.881903 16912 raft_consensus.cc:348] T 00000000000000000000000000000000 P d47abc77ad394d25b7f378cbb1ef4a60 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d47abc77ad394d25b7f378cbb1ef4a60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39645 } }
I20260925 16:36:33.882300 16912 raft_consensus.cc:729] T 00000000000000000000000000000000 P d47abc77ad394d25b7f378cbb1ef4a60 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d47abc77ad394d25b7f378cbb1ef4a60, State: Initialized, Role: FOLLOWER
I20260925 16:36:33.882850 16912 consensus_queue.cc:260] T 00000000000000000000000000000000 P d47abc77ad394d25b7f378cbb1ef4a60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d47abc77ad394d25b7f378cbb1ef4a60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39645 } }
I20260925 16:36:33.882997 16912 raft_consensus.cc:388] T 00000000000000000000000000000000 P d47abc77ad394d25b7f378cbb1ef4a60 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:36:33.883086 16912 raft_consensus.cc:482] T 00000000000000000000000000000000 P d47abc77ad394d25b7f378cbb1ef4a60 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:36:33.883203 16912 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d47abc77ad394d25b7f378cbb1ef4a60 [term 2 FOLLOWER]: Advancing to term 3
I20260925 16:36:33.885123 16912 raft_consensus.cc:504] T 00000000000000000000000000000000 P d47abc77ad394d25b7f378cbb1ef4a60 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d47abc77ad394d25b7f378cbb1ef4a60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39645 } }
I20260925 16:36:33.885478 16912 leader_election.cc:302] T 00000000000000000000000000000000 P d47abc77ad394d25b7f378cbb1ef4a60 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d47abc77ad394d25b7f378cbb1ef4a60; no voters:
I20260925 16:36:33.885782 16912 leader_election.cc:288] T 00000000000000000000000000000000 P d47abc77ad394d25b7f378cbb1ef4a60 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 16:36:33.885918 16916 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d47abc77ad394d25b7f378cbb1ef4a60 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 16:36:33.886183 16916 raft_consensus.cc:686] T 00000000000000000000000000000000 P d47abc77ad394d25b7f378cbb1ef4a60 [term 3 LEADER]: Becoming Leader. State: Replica: d47abc77ad394d25b7f378cbb1ef4a60, State: Running, Role: LEADER
I20260925 16:36:33.886561 16916 consensus_queue.cc:237] T 00000000000000000000000000000000 P d47abc77ad394d25b7f378cbb1ef4a60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d47abc77ad394d25b7f378cbb1ef4a60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39645 } }
I20260925 16:36:33.887039 16912 sys_catalog.cc:564] T 00000000000000000000000000000000 P d47abc77ad394d25b7f378cbb1ef4a60 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:36:33.888024 16917 sys_catalog.cc:455] T 00000000000000000000000000000000 P d47abc77ad394d25b7f378cbb1ef4a60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "d47abc77ad394d25b7f378cbb1ef4a60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d47abc77ad394d25b7f378cbb1ef4a60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39645 } } }
I20260925 16:36:33.888141 16917 sys_catalog.cc:458] T 00000000000000000000000000000000 P d47abc77ad394d25b7f378cbb1ef4a60 [sys.catalog]: This master's current role is: LEADER
I20260925 16:36:33.888443 16918 sys_catalog.cc:455] T 00000000000000000000000000000000 P d47abc77ad394d25b7f378cbb1ef4a60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d47abc77ad394d25b7f378cbb1ef4a60. Latest consensus state: current_term: 3 leader_uuid: "d47abc77ad394d25b7f378cbb1ef4a60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d47abc77ad394d25b7f378cbb1ef4a60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39645 } } }
I20260925 16:36:33.888579 16918 sys_catalog.cc:458] T 00000000000000000000000000000000 P d47abc77ad394d25b7f378cbb1ef4a60 [sys.catalog]: This master's current role is: LEADER
I20260925 16:36:33.888863 16925 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:36:33.894116 16925 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:36:33.894769 16925 catalog_manager.cc:1259] Loaded cluster ID: b54e38f54c674aca97a7b343c71948b4
I20260925 16:36:33.894830 16925 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:36:33.897621 16925 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:36:33.898321 16925 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d47abc77ad394d25b7f378cbb1ef4a60: Loaded TSK: 0
I20260925 16:36:33.898403 16925 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d47abc77ad394d25b7f378cbb1ef4a60: Loaded TSK: 1
F20260925 16:36:33.899121 16925 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f21e1c409ad google::LogMessage::Fail() at ??:0
@ 0x7f21e1c4388a google::LogMessage::SendToLog() at ??:0
@ 0x7f21e1c404de google::LogMessage::Flush() at ??:0
@ 0x7f21e1c41a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f21e490418d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f21e48fcf0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f21ec5a57bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f21ec57f99c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f21ec580178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f21ec5c1df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f21ec5e082b std::function<>::operator()() at ??:0
@ 0x7f21ec57fb4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f21ec580f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f21ec57d579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f21ec5c1640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f21e3e923d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f21e3e92c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f21e3e943f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f21e3e83512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f21e54c76db start_thread at ??:0
@ 0x7f21e073e61f clone at ??:0
W20260925 16:36:34.002880 16914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:36:34.003129 16914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:36:34.003171 16914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:36:34.007287 16914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:36:34.007346 16914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:36:34.007526 16914 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 16:36:34.012964 16914 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:37719
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=45121
--tserver_master_addrs=127.0.98.62:39645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16914
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:36:34.014051 16914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:36:34.015244 16914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:36:34.025013 16959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:36:34.025085 16960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:36:34.025013 16962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:36:34.025355 16914 server_base.cc:1034] running on GCE node
I20260925 16:36:34.025790 16914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:36:34.026432 16914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:36:34.027623 16914 hybrid_clock.cc:648] HybridClock initialized: now 1790354194027608 us; error 36 us; skew 500 ppm
Sep 25 16:36:34 dist-test-slave-65pf krb5kdc[15769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354194, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:36:34.030337 16914 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 16:36:34.031245 16914 webserver.cc:460] Webserver started at http://127.0.98.1:45121/ using document root <none> and password file <none>
I20260925 16:36:34.031824 16914 fs_manager.cc:359] Metadata directory not provided
I20260925 16:36:34.031885 16914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:36:34.035739 16914 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 16:36:34.038028 16969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:34.039337 16914 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 16:36:34.039470 16914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-0/wal
uuid: "fd2d720c99f84f098a1054646aedd2a9"
format_stamp: "Formatted at 2026-09-25 16:35:51 on dist-test-slave-65pf"
server_key: "600604f57e929abf76e71ef83692abb6"
server_key_iv: "cb60f45afaf6022379f1a9fedb1ace19"
server_key_version: "encryptionkey@0"
I20260925 16:36:34.040042 16914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:36:34.056133 16914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:36:34.059643 16914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:36:34.059857 16914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:36:34.060428 16914 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:36:34.061543 16976 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 16:36:34.067067 16914 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 16:36:34.067149 16914 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260925 16:36:34.067252 16914 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 16:36:34.068953 16914 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 16:36:34.069024 16914 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260925 16:36:34.069319 16976 tablet_bootstrap.cc:492] T 2e8df4ca01f74f568b40105f5258ae87 P fd2d720c99f84f098a1054646aedd2a9: Bootstrap starting.
I20260925 16:36:34.079522 16914 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37719
I20260925 16:36:34.079770 17083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37719 every 8 connection(s)
I20260925 16:36:34.080551 16914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
I20260925 16:36:34.087831 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 16914
I20260925 16:36:34.088285 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:43913
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:39645
--webserver_port=36163
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 16:36:34.091382 16976 log.cc:824] T 2e8df4ca01f74f568b40105f5258ae87 P fd2d720c99f84f098a1054646aedd2a9: Log is configured to *not* fsync() on all Append() calls
I20260925 16:36:34.100649 16976 tablet_bootstrap.cc:492] T 2e8df4ca01f74f568b40105f5258ae87 P fd2d720c99f84f098a1054646aedd2a9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 16:36:34.101045 16976 tablet_bootstrap.cc:492] T 2e8df4ca01f74f568b40105f5258ae87 P fd2d720c99f84f098a1054646aedd2a9: Bootstrap complete.
I20260925 16:36:34.102483 16976 ts_tablet_manager.cc:1397] T 2e8df4ca01f74f568b40105f5258ae87 P fd2d720c99f84f098a1054646aedd2a9: Time spent bootstrapping tablet: real 0.033s user 0.025s sys 0.004s
I20260925 16:36:34.105666 16976 raft_consensus.cc:348] T 2e8df4ca01f74f568b40105f5258ae87 P fd2d720c99f84f098a1054646aedd2a9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95b51fa015394f2e892650b4f9e1bfd2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } } peers { permanent_uuid: "a114478b9c5b4e14a0de2caf167fe07b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43913 } } peers { permanent_uuid: "fd2d720c99f84f098a1054646aedd2a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37719 } }
I20260925 16:36:34.106194 16976 raft_consensus.cc:729] T 2e8df4ca01f74f568b40105f5258ae87 P fd2d720c99f84f098a1054646aedd2a9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd2d720c99f84f098a1054646aedd2a9, State: Initialized, Role: FOLLOWER
I20260925 16:36:34.106781 16976 consensus_queue.cc:260] T 2e8df4ca01f74f568b40105f5258ae87 P fd2d720c99f84f098a1054646aedd2a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95b51fa015394f2e892650b4f9e1bfd2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } } peers { permanent_uuid: "a114478b9c5b4e14a0de2caf167fe07b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43913 } } peers { permanent_uuid: "fd2d720c99f84f098a1054646aedd2a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37719 } }
I20260925 16:36:34.107903 16976 ts_tablet_manager.cc:1428] T 2e8df4ca01f74f568b40105f5258ae87 P fd2d720c99f84f098a1054646aedd2a9: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
W20260925 16:36:34.125661 17086 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:36:34.082469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39645 (local address 127.0.98.1:35257)
0925 16:36:34.083000 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:34.083055 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 16:36:34.083874 (+ 819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:36:34.124884 (+ 41010us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39645: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":192,"threads_started":1}
W20260925 16:36:34.126174 17084 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39645 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:39645: Client connection negotiation failed: client connection to 127.0.98.62:39645: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 16:36:34.126951 17086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:34.126715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39645 (local address 127.0.98.1:49975)
0925 16:36:34.126838 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:34.126898 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 16:36:34.127633 17086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:34.127460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39645 (local address 127.0.98.1:53971)
0925 16:36:34.127553 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:34.127591 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260925 16:36:34.127847 17084 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 16:36:34.202569 17087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:36:34.202841 17087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:36:34.202924 17087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:36:34.206930 17087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:36:34.206990 17087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:36:34.207087 17087 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 16:36:34.212227 17087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:43913
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36163
--tserver_master_addrs=127.0.98.62:39645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:36:34.213280 17087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:36:34.214327 17087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:36:34.224398 17095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:36:34.224414 17094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:36:34.224421 17097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:36:34.224673 17087 server_base.cc:1034] running on GCE node
I20260925 16:36:34.225049 17087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:36:34.225682 17087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:36:34.226872 17087 hybrid_clock.cc:648] HybridClock initialized: now 1790354194226859 us; error 28 us; skew 500 ppm
Sep 25 16:36:34 dist-test-slave-65pf krb5kdc[15769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354194, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:36:34.229647 17087 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 16:36:34.230466 17087 webserver.cc:460] Webserver started at http://127.0.98.2:36163/ using document root <none> and password file <none>
I20260925 16:36:34.231045 17087 fs_manager.cc:359] Metadata directory not provided
I20260925 16:36:34.231106 17087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:36:34.234828 17087 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260925 16:36:34.236780 17104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:34.237977 17087 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:36:34.238102 17087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-1/wal
uuid: "a114478b9c5b4e14a0de2caf167fe07b"
format_stamp: "Formatted at 2026-09-25 16:35:51 on dist-test-slave-65pf"
server_key: "fdb9f4d9226fa4df152bf958bbdfe358"
server_key_iv: "daf8c7d45243c5ed6221e8d143baadf0"
server_key_version: "encryptionkey@0"
I20260925 16:36:34.238715 17087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:36:34.263298 17087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:36:34.266682 17087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:36:34.266901 17087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:36:34.267493 17087 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:36:34.268333 17087 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:36:34.268381 17087 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:34.268451 17087 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:36:34.268488 17087 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:34.276969 17087 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43913
I20260925 16:36:34.277043 17217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43913 every 8 connection(s)
I20260925 16:36:34.278000 17087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
I20260925 16:36:34.281996 17220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:34.280121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39645 (local address 127.0.98.2:60867)
0925 16:36:34.280640 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:34.281246 (+ 606us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":110,"threads_started":1}
W20260925 16:36:34.282442 17218 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39645 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:39645: Client connection negotiation failed: client connection to 127.0.98.62:39645: connect: Connection refused (error 111)
I20260925 16:36:34.283004 17220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:34.282804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39645 (local address 127.0.98.2:53589)
0925 16:36:34.282906 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:34.282951 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 16:36:34.283664 17220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:34.283444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39645 (local address 127.0.98.2:33357)
0925 16:36:34.283562 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:34.283616 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260925 16:36:34.283864 17218 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 16:36:34.284842 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 17087
I20260925 16:36:34.285220 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:37195
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:39645
--webserver_port=33447
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:36:34.398648 17221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:36:34.398978 17221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:36:34.399053 17221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:36:34.403200 17221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:36:34.403286 17221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:36:34.403376 17221 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 16:36:34.408601 17221 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:37195
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=33447
--tserver_master_addrs=127.0.98.62:39645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17221
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:36:34.409713 17221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:36:34.410836 17221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:36:34.420277 17227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:36:34.420288 17226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:36:34.420629 17221 server_base.cc:1034] running on GCE node
W20260925 16:36:34.420697 17229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:36:34.421209 17221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:36:34.421892 17221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:36:34.423048 17221 hybrid_clock.cc:648] HybridClock initialized: now 1790354194423034 us; error 26 us; skew 500 ppm
Sep 25 16:36:34 dist-test-slave-65pf krb5kdc[15769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354194, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:36:34.425786 17221 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 16:36:34.426715 17221 webserver.cc:460] Webserver started at http://127.0.98.3:33447/ using document root <none> and password file <none>
I20260925 16:36:34.427232 17221 fs_manager.cc:359] Metadata directory not provided
I20260925 16:36:34.427290 17221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:36:34.430970 17221 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260925 16:36:34.432924 17236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:34.434122 17221 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:36:34.434211 17221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-2/wal
uuid: "95b51fa015394f2e892650b4f9e1bfd2"
format_stamp: "Formatted at 2026-09-25 16:35:52 on dist-test-slave-65pf"
server_key: "1d2c2f1d0f4729f800dcafd2508a7234"
server_key_iv: "072658af143c19ca82e2fda2a4849fd3"
server_key_version: "encryptionkey@0"
I20260925 16:36:34.434793 17221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:36:34.446952 17221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:36:34.450347 17221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:36:34.450519 17221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:36:34.451176 17221 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:36:34.452018 17221 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:36:34.452068 17221 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:34.452112 17221 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:36:34.452131 17221 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:34.462088 17221 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37195
I20260925 16:36:34.462108 17349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37195 every 8 connection(s)
I20260925 16:36:34.463214 17221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
I20260925 16:36:34.467203 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:34.465521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39645 (local address 127.0.98.3:52723)
0925 16:36:34.466002 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:34.466482 (+ 480us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":111,"threads_started":1}
W20260925 16:36:34.467671 17350 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39645 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:39645: Client connection negotiation failed: client connection to 127.0.98.62:39645: connect: Connection refused (error 111)
I20260925 16:36:34.468281 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:34.468065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39645 (local address 127.0.98.3:35475)
0925 16:36:34.468176 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:34.468230 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 16:36:34.468982 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:34.468697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39645 (local address 127.0.98.3:53715)
0925 16:36:34.468849 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:34.468916 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 16:36:34.469216 17350 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 16:36:34.471437 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 17221
I20260925 16:36:35.129004 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:35.128494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39645 (local address 127.0.98.1:54733)
0925 16:36:35.128802 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:35.128862 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":114,"threads_started":1}
I20260925 16:36:35.254426 17355 raft_consensus.cc:482] T 2e8df4ca01f74f568b40105f5258ae87 P fd2d720c99f84f098a1054646aedd2a9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:36:35.254688 17355 raft_consensus.cc:504] T 2e8df4ca01f74f568b40105f5258ae87 P fd2d720c99f84f098a1054646aedd2a9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95b51fa015394f2e892650b4f9e1bfd2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } } peers { permanent_uuid: "a114478b9c5b4e14a0de2caf167fe07b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43913 } } peers { permanent_uuid: "fd2d720c99f84f098a1054646aedd2a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37719 } }
I20260925 16:36:35.255882 17355 leader_election.cc:288] T 2e8df4ca01f74f568b40105f5258ae87 P fd2d720c99f84f098a1054646aedd2a9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 95b51fa015394f2e892650b4f9e1bfd2 (127.0.98.3:37195), a114478b9c5b4e14a0de2caf167fe07b (127.0.98.2:43913)
Sep 25 16:36:35 dist-test-slave-65pf krb5kdc[15769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354194, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 25 16:36:35 dist-test-slave-65pf krb5kdc[15769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354194, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260925 16:36:35.269958 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:35.256113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37195 (local address 127.0.98.1:51155)
0925 16:36:35.256225 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:35.256241 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:36:35.256353 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:36:35.258109 (+ 1756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:36:35.258115 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:36:35.258702 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:36:35.259214 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:35.259232 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:35.260787 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:35.260790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:35.261685 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:35.261695 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:35.262404 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:35.262410 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:35.262607 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:35.263293 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:36:35.263309 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:36:35.265413 (+ 2104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:36:35.267511 (+ 2098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:36:35.267514 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:36:35.267515 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:36:35.267767 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:36:35.268122 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:36:35.268125 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:36:35.268127 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:36:35.268296 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:36:35.268940 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:36:35.268946 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:36:35.269210 (+ 264us) client_negotiation.cc:769] Sending connection context
0925 16:36:35.269400 (+ 190us) client_negotiation.cc:240] Negotiation successful
0925 16:36:35.269769 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":128,"spinlock_wait_cycles":19840}
I20260925 16:36:35.269972 17358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:35.256581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34035 (local address 127.0.98.2:43913)
0925 16:36:35.257027 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:35.257036 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:35.257066 (+ 30us) server_negotiation.cc:407] Connection header received
0925 16:36:35.257678 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:35.257683 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:35.258042 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:35.258343 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:36:35.259391 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:35.260610 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:35.261814 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:35.262302 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:35.262484 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:35.265189 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:36:35.265196 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:36:35.265213 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:36:35.265248 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:36:35.267299 (+ 2051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:36:35.267878 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:36:35.267885 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:36:35.267891 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:36:35.268015 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:36:35.268386 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:36:35.268389 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:36:35.268391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:36:35.268828 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:36:35.268973 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:35.269524 (+ 551us) server_negotiation.cc:299] Negotiation successful
0925 16:36:35.269784 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":115,"threads_started":1}
I20260925 16:36:35.269999 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:35.256280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51155 (local address 127.0.98.3:37195)
0925 16:36:35.256651 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:35.256664 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:35.256702 (+ 38us) server_negotiation.cc:407] Connection header received
0925 16:36:35.257380 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:35.257386 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:35.257752 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:35.257997 (+ 245us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:36:35.259403 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:35.260630 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:35.261816 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:35.262282 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:35.262440 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:35.265632 (+ 3192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:36:35.265638 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:36:35.265649 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:36:35.265678 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:36:35.267393 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:36:35.267902 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:36:35.267907 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:36:35.267911 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:36:35.268019 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:36:35.268414 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:36:35.268417 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:36:35.268418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:36:35.268818 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:36:35.268952 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:35.269524 (+ 572us) server_negotiation.cc:299] Negotiation successful
0925 16:36:35.269814 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":115,"threads_started":1}
I20260925 16:36:35.270015 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:35.256404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43913 (local address 127.0.98.1:34035)
0925 16:36:35.256715 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:35.256774 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 16:36:35.256887 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:36:35.258298 (+ 1411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:36:35.258301 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:36:35.258820 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:36:35.259224 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:35.259232 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:35.260748 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:35.260751 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:35.261687 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:35.261696 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:35.262455 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:35.262459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:35.262617 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:35.263344 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:36:35.263355 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:36:35.264925 (+ 1570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:36:35.267418 (+ 2493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:36:35.267426 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:36:35.267443 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:36:35.267762 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:36:35.268118 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:36:35.268122 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:36:35.268124 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:36:35.268289 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:36:35.268962 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:36:35.268968 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:36:35.269221 (+ 253us) client_negotiation.cc:769] Sending connection context
0925 16:36:35.269400 (+ 179us) client_negotiation.cc:240] Negotiation successful
0925 16:36:35.269846 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":330,"spinlock_wait_cycles":46464,"thread_start_us":87,"threads_started":1}
W20260925 16:36:35.272907 16970 leader_election.cc:341] T 2e8df4ca01f74f568b40105f5258ae87 P fd2d720c99f84f098a1054646aedd2a9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 95b51fa015394f2e892650b4f9e1bfd2 (127.0.98.3:37195): Not found: Tablet not found: 2e8df4ca01f74f568b40105f5258ae87
W20260925 16:36:35.273092 16970 leader_election.cc:341] T 2e8df4ca01f74f568b40105f5258ae87 P fd2d720c99f84f098a1054646aedd2a9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a114478b9c5b4e14a0de2caf167fe07b (127.0.98.2:43913): Not found: Tablet not found: 2e8df4ca01f74f568b40105f5258ae87
I20260925 16:36:35.273137 16970 leader_election.cc:302] T 2e8df4ca01f74f568b40105f5258ae87 P fd2d720c99f84f098a1054646aedd2a9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd2d720c99f84f098a1054646aedd2a9; no voters: 95b51fa015394f2e892650b4f9e1bfd2, a114478b9c5b4e14a0de2caf167fe07b
I20260925 16:36:35.273371 17355 raft_consensus.cc:2726] T 2e8df4ca01f74f568b40105f5258ae87 P fd2d720c99f84f098a1054646aedd2a9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 16:36:35.284825 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:35.284418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39645 (local address 127.0.98.2:38419)
0925 16:36:35.284644 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:35.284705 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":77,"threads_started":1}
I20260925 16:36:35.470129 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:35.469756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39645 (local address 127.0.98.3:36985)
0925 16:36:35.469982 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:35.470035 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":70,"threads_started":1}
I20260925 16:36:36.130328 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:36.129872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39645 (local address 127.0.98.1:56809)
0925 16:36:36.130165 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:36.130229 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":83,"threads_started":1}
I20260925 16:36:36.286053 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:36.285543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39645 (local address 127.0.98.2:49629)
0925 16:36:36.285877 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:36.285947 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":71,"threads_started":1}
I20260925 16:36:36.471442 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:36.471017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39645 (local address 127.0.98.3:39751)
0925 16:36:36.471277 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:36.471333 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":71,"threads_started":1}
I20260925 16:36:37.131510 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:37.131051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39645 (local address 127.0.98.1:41041)
0925 16:36:37.131340 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:37.131405 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":69,"threads_started":1}
I20260925 16:36:37.287310 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:37.286888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39645 (local address 127.0.98.2:47655)
0925 16:36:37.287150 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:37.287208 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":85,"threads_started":1}
I20260925 16:36:37.387899 17366 raft_consensus.cc:482] T 2e8df4ca01f74f568b40105f5258ae87 P fd2d720c99f84f098a1054646aedd2a9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:36:37.388055 17366 raft_consensus.cc:504] T 2e8df4ca01f74f568b40105f5258ae87 P fd2d720c99f84f098a1054646aedd2a9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95b51fa015394f2e892650b4f9e1bfd2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } } peers { permanent_uuid: "a114478b9c5b4e14a0de2caf167fe07b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43913 } } peers { permanent_uuid: "fd2d720c99f84f098a1054646aedd2a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37719 } }
I20260925 16:36:37.388686 17366 leader_election.cc:288] T 2e8df4ca01f74f568b40105f5258ae87 P fd2d720c99f84f098a1054646aedd2a9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 95b51fa015394f2e892650b4f9e1bfd2 (127.0.98.3:37195), a114478b9c5b4e14a0de2caf167fe07b (127.0.98.2:43913)
W20260925 16:36:37.389603 16970 leader_election.cc:341] T 2e8df4ca01f74f568b40105f5258ae87 P fd2d720c99f84f098a1054646aedd2a9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 95b51fa015394f2e892650b4f9e1bfd2 (127.0.98.3:37195): Not found: Tablet not found: 2e8df4ca01f74f568b40105f5258ae87
W20260925 16:36:37.389729 16970 leader_election.cc:341] T 2e8df4ca01f74f568b40105f5258ae87 P fd2d720c99f84f098a1054646aedd2a9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a114478b9c5b4e14a0de2caf167fe07b (127.0.98.2:43913): Not found: Tablet not found: 2e8df4ca01f74f568b40105f5258ae87
I20260925 16:36:37.389810 16970 leader_election.cc:302] T 2e8df4ca01f74f568b40105f5258ae87 P fd2d720c99f84f098a1054646aedd2a9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd2d720c99f84f098a1054646aedd2a9; no voters: 95b51fa015394f2e892650b4f9e1bfd2, a114478b9c5b4e14a0de2caf167fe07b
I20260925 16:36:37.389988 17366 raft_consensus.cc:2726] T 2e8df4ca01f74f568b40105f5258ae87 P fd2d720c99f84f098a1054646aedd2a9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 16:36:37.472666 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:37.472248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39645 (local address 127.0.98.3:58321)
0925 16:36:37.472521 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:37.472575 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":93,"threads_started":1}
I20260925 16:36:38.132862 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:38.132421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39645 (local address 127.0.98.1:51115)
0925 16:36:38.132691 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:38.132767 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":89,"threads_started":1}
I20260925 16:36:38.288447 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:38.288058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39645 (local address 127.0.98.2:38983)
0925 16:36:38.288304 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:38.288357 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":83,"threads_started":1}
I20260925 16:36:38.474014 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:38.473584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39645 (local address 127.0.98.3:56067)
0925 16:36:38.473864 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:38.473920 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":84,"threads_started":1}
I20260925 16:36:39.134392 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:39.133946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39645 (local address 127.0.98.1:49859)
0925 16:36:39.134232 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:39.134292 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":71,"threads_started":1}
I20260925 16:36:39.219954 17372 raft_consensus.cc:482] T 2e8df4ca01f74f568b40105f5258ae87 P fd2d720c99f84f098a1054646aedd2a9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:36:39.220125 17372 raft_consensus.cc:504] T 2e8df4ca01f74f568b40105f5258ae87 P fd2d720c99f84f098a1054646aedd2a9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95b51fa015394f2e892650b4f9e1bfd2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } } peers { permanent_uuid: "a114478b9c5b4e14a0de2caf167fe07b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43913 } } peers { permanent_uuid: "fd2d720c99f84f098a1054646aedd2a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37719 } }
I20260925 16:36:39.220551 17372 leader_election.cc:288] T 2e8df4ca01f74f568b40105f5258ae87 P fd2d720c99f84f098a1054646aedd2a9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 95b51fa015394f2e892650b4f9e1bfd2 (127.0.98.3:37195), a114478b9c5b4e14a0de2caf167fe07b (127.0.98.2:43913)
W20260925 16:36:39.221421 16970 leader_election.cc:341] T 2e8df4ca01f74f568b40105f5258ae87 P fd2d720c99f84f098a1054646aedd2a9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a114478b9c5b4e14a0de2caf167fe07b (127.0.98.2:43913): Not found: Tablet not found: 2e8df4ca01f74f568b40105f5258ae87
W20260925 16:36:39.221552 16970 leader_election.cc:341] T 2e8df4ca01f74f568b40105f5258ae87 P fd2d720c99f84f098a1054646aedd2a9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 95b51fa015394f2e892650b4f9e1bfd2 (127.0.98.3:37195): Not found: Tablet not found: 2e8df4ca01f74f568b40105f5258ae87
I20260925 16:36:39.221608 16970 leader_election.cc:302] T 2e8df4ca01f74f568b40105f5258ae87 P fd2d720c99f84f098a1054646aedd2a9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd2d720c99f84f098a1054646aedd2a9; no voters: 95b51fa015394f2e892650b4f9e1bfd2, a114478b9c5b4e14a0de2caf167fe07b
I20260925 16:36:39.221812 17372 raft_consensus.cc:2726] T 2e8df4ca01f74f568b40105f5258ae87 P fd2d720c99f84f098a1054646aedd2a9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 16:36:39.289742 17373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:39.289305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39645 (local address 127.0.98.2:41251)
0925 16:36:39.289573 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:39.289637 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":88,"threads_started":1}
I20260925 16:36:39.475418 17374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:39.474867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39645 (local address 127.0.98.3:56803)
0925 16:36:39.475180 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:39.475280 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":127,"threads_started":1}
I20260925 16:36:40.135555 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:40.135148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39645 (local address 127.0.98.1:43987)
0925 16:36:40.135400 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:40.135457 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":84,"threads_started":1}
I20260925 16:36:40.290977 17376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:40.290473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39645 (local address 127.0.98.2:38773)
0925 16:36:40.290818 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:40.290879 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":127,"threads_started":1}
I20260925 16:36:40.476747 17377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:40.476321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39645 (local address 127.0.98.3:41159)
0925 16:36:40.476594 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:40.476650 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":106,"threads_started":1}
I20260925 16:36:41.136729 17378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:41.136340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39645 (local address 127.0.98.1:51799)
0925 16:36:41.136583 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:41.136635 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":73,"threads_started":1}
I20260925 16:36:41.186290 17379 raft_consensus.cc:482] T 2e8df4ca01f74f568b40105f5258ae87 P fd2d720c99f84f098a1054646aedd2a9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:36:41.186422 17379 raft_consensus.cc:504] T 2e8df4ca01f74f568b40105f5258ae87 P fd2d720c99f84f098a1054646aedd2a9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95b51fa015394f2e892650b4f9e1bfd2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } } peers { permanent_uuid: "a114478b9c5b4e14a0de2caf167fe07b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43913 } } peers { permanent_uuid: "fd2d720c99f84f098a1054646aedd2a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37719 } }
I20260925 16:36:41.186949 17379 leader_election.cc:288] T 2e8df4ca01f74f568b40105f5258ae87 P fd2d720c99f84f098a1054646aedd2a9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 95b51fa015394f2e892650b4f9e1bfd2 (127.0.98.3:37195), a114478b9c5b4e14a0de2caf167fe07b (127.0.98.2:43913)
W20260925 16:36:41.187737 16970 leader_election.cc:341] T 2e8df4ca01f74f568b40105f5258ae87 P fd2d720c99f84f098a1054646aedd2a9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 95b51fa015394f2e892650b4f9e1bfd2 (127.0.98.3:37195): Not found: Tablet not found: 2e8df4ca01f74f568b40105f5258ae87
W20260925 16:36:41.187909 16970 leader_election.cc:341] T 2e8df4ca01f74f568b40105f5258ae87 P fd2d720c99f84f098a1054646aedd2a9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a114478b9c5b4e14a0de2caf167fe07b (127.0.98.2:43913): Not found: Tablet not found: 2e8df4ca01f74f568b40105f5258ae87
I20260925 16:36:41.187968 16970 leader_election.cc:302] T 2e8df4ca01f74f568b40105f5258ae87 P fd2d720c99f84f098a1054646aedd2a9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd2d720c99f84f098a1054646aedd2a9; no voters: 95b51fa015394f2e892650b4f9e1bfd2, a114478b9c5b4e14a0de2caf167fe07b
I20260925 16:36:41.188148 17379 raft_consensus.cc:2726] T 2e8df4ca01f74f568b40105f5258ae87 P fd2d720c99f84f098a1054646aedd2a9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 16:36:41.292196 17380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:41.291783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39645 (local address 127.0.98.2:44641)
0925 16:36:41.292047 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:41.292100 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":71,"threads_started":1}
I20260925 16:36:41.477977 17381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:41.477512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39645 (local address 127.0.98.3:58111)
0925 16:36:41.477820 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:41.477878 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":81,"threads_started":1}
I20260925 16:36:42.138249 17382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:42.137736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39645 (local address 127.0.98.1:43941)
0925 16:36:42.138034 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:42.138091 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":96,"threads_started":1}
I20260925 16:36:42.293330 17383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:42.292933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39645 (local address 127.0.98.2:56649)
0925 16:36:42.293169 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:42.293224 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":72,"threads_started":1}
I20260925 16:36:42.479195 17384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:42.478768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39645 (local address 127.0.98.3:36033)
0925 16:36:42.479042 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:42.479099 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":79,"threads_started":1}
I20260925 16:36:43.138244 17385 raft_consensus.cc:482] T 2e8df4ca01f74f568b40105f5258ae87 P fd2d720c99f84f098a1054646aedd2a9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:36:43.138401 17385 raft_consensus.cc:504] T 2e8df4ca01f74f568b40105f5258ae87 P fd2d720c99f84f098a1054646aedd2a9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95b51fa015394f2e892650b4f9e1bfd2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } } peers { permanent_uuid: "a114478b9c5b4e14a0de2caf167fe07b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43913 } } peers { permanent_uuid: "fd2d720c99f84f098a1054646aedd2a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37719 } }
I20260925 16:36:43.138896 17385 leader_election.cc:288] T 2e8df4ca01f74f568b40105f5258ae87 P fd2d720c99f84f098a1054646aedd2a9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 95b51fa015394f2e892650b4f9e1bfd2 (127.0.98.3:37195), a114478b9c5b4e14a0de2caf167fe07b (127.0.98.2:43913)
I20260925 16:36:43.139415 17386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:43.139001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39645 (local address 127.0.98.1:54465)
0925 16:36:43.139247 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:43.139313 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":67,"threads_started":1}
W20260925 16:36:43.139607 16970 leader_election.cc:341] T 2e8df4ca01f74f568b40105f5258ae87 P fd2d720c99f84f098a1054646aedd2a9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 95b51fa015394f2e892650b4f9e1bfd2 (127.0.98.3:37195): Not found: Tablet not found: 2e8df4ca01f74f568b40105f5258ae87
W20260925 16:36:43.139745 16970 leader_election.cc:341] T 2e8df4ca01f74f568b40105f5258ae87 P fd2d720c99f84f098a1054646aedd2a9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a114478b9c5b4e14a0de2caf167fe07b (127.0.98.2:43913): Not found: Tablet not found: 2e8df4ca01f74f568b40105f5258ae87
I20260925 16:36:43.139788 16970 leader_election.cc:302] T 2e8df4ca01f74f568b40105f5258ae87 P fd2d720c99f84f098a1054646aedd2a9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd2d720c99f84f098a1054646aedd2a9; no voters: 95b51fa015394f2e892650b4f9e1bfd2, a114478b9c5b4e14a0de2caf167fe07b
I20260925 16:36:43.139910 17385 raft_consensus.cc:2726] T 2e8df4ca01f74f568b40105f5258ae87 P fd2d720c99f84f098a1054646aedd2a9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 16:36:43.294437 17387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:43.294013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39645 (local address 127.0.98.2:38193)
0925 16:36:43.294291 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:43.294345 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":73,"threads_started":1}
I20260925 16:36:43.480330 17388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:43.479933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39645 (local address 127.0.98.3:52327)
0925 16:36:43.480174 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:43.480227 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":80,"threads_started":1}
I20260925 16:36:44.140743 17389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:44.140324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39645 (local address 127.0.98.1:36097)
0925 16:36:44.140571 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:44.140630 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":76,"threads_started":1}
I20260925 16:36:44.295760 17390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:44.295320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39645 (local address 127.0.98.2:39527)
0925 16:36:44.295591 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:44.295655 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":75,"threads_started":1}
I20260925 16:36:44.481772 17391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:44.481291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39645 (local address 127.0.98.3:41041)
0925 16:36:44.481563 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:44.481634 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":79,"threads_started":1}
I20260925 16:36:45.141957 17392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:45.141545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39645 (local address 127.0.98.1:41153)
0925 16:36:45.141800 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:45.141861 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":80,"threads_started":1}
I20260925 16:36:45.296959 17393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:45.296534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39645 (local address 127.0.98.2:52161)
0925 16:36:45.296803 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:45.296860 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":84,"threads_started":1}
I20260925 16:36:45.483002 17394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:45.482536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39645 (local address 127.0.98.3:39173)
0925 16:36:45.482846 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:45.482902 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":141,"threads_started":1}
I20260925 16:36:46.143115 17395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:46.142721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39645 (local address 127.0.98.1:53921)
0925 16:36:46.142963 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:46.143018 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":105,"threads_started":1}
I20260925 16:36:46.298194 17396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:46.297743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39645 (local address 127.0.98.2:43321)
0925 16:36:46.298041 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:46.298099 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":101,"threads_started":1}
I20260925 16:36:46.484265 17397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:46.483749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39645 (local address 127.0.98.3:42321)
0925 16:36:46.484082 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:46.484150 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":87,"threads_started":1}
I20260925 16:36:47.144315 17398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:47.143916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39645 (local address 127.0.98.1:47729)
0925 16:36:47.144162 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:47.144218 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":90,"threads_started":1}
I20260925 16:36:47.299489 17399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:47.299060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39645 (local address 127.0.98.2:43909)
0925 16:36:47.299333 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:47.299391 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":89,"threads_started":1}
I20260925 16:36:47.485536 17400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:47.485105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39645 (local address 127.0.98.3:41261)
0925 16:36:47.485372 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:47.485430 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":84,"threads_started":1}
I20260925 16:36:48.145514 17401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:48.145070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39645 (local address 127.0.98.1:44591)
0925 16:36:48.145341 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:48.145404 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":73,"threads_started":1}
I20260925 16:36:48.300799 17402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:48.300341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39645 (local address 127.0.98.2:47081)
0925 16:36:48.300614 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:48.300672 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":82,"threads_started":1}
I20260925 16:36:48.486829 17403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:48.486313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39645 (local address 127.0.98.3:37421)
0925 16:36:48.486660 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:48.486731 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":83,"threads_started":1}
I20260925 16:36:49.146872 17404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:49.146381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39645 (local address 127.0.98.1:54481)
0925 16:36:49.146697 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:49.146774 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":79,"threads_started":1}
I20260925 16:36:49.302129 17405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:49.301663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39645 (local address 127.0.98.2:51661)
0925 16:36:49.301946 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:49.302020 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":73,"threads_started":1}
I20260925 16:36:49.481935 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 16914
I20260925 16:36:49.487864 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 17087
I20260925 16:36:49.488521 17406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:49.487709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39645 (local address 127.0.98.3:45835)
0925 16:36:49.487950 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:49.488382 (+ 432us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":91,"threads_started":1}
I20260925 16:36:49.493346 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 17221
2026-09-25T16:36:49Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59304 ms)
[----------] 30 tests from SecurityITest (242670 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:36:49 dist-test-slave-65pf krb5kdc[17409](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:36:49 dist-test-slave-65pf krb5kdc[17409](info): set up 2 sockets
Sep 25 16:36:49 dist-test-slave-65pf krb5kdc[17409](info): commencing operation
krb5kdc: starting...
W20260925 16:36:50.538535 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/krb5kdc/test-user.keytab.
Sep 25 16:36:50 dist-test-slave-65pf krb5kdc[17409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354210, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:36:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:36:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 16:36:50.665409 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33759
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33759
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:36:50.772809 17422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:36:50.773056 17422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:36:50.773092 17422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:36:50.777070 17422 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:36:50.777125 17422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:36:50.777144 17422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:36:50.777161 17422 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:36:50.777235 17422 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:36:50.782253 17422 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33759
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17422
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:36:50.783288 17422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:36:50.784274 17422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 16:36:50.792872 17422 server_base.cc:1034] running on GCE node
W20260925 16:36:50.792793 17427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:36:50.792896 17430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:36:50.792793 17428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:36:50.793781 17422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:36:50.794875 17422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:36:50.796025 17422 hybrid_clock.cc:648] HybridClock initialized: now 1790354210796009 us; error 37 us; skew 500 ppm
Sep 25 16:36:50 dist-test-slave-65pf krb5kdc[17409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354210, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:36:50.798702 17422 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 16:36:50.799520 17422 webserver.cc:460] Webserver started at http://127.0.98.62:34881/ using document root <none> and password file <none>
I20260925 16:36:50.800006 17422 fs_manager.cc:359] Metadata directory not provided
I20260925 16:36:50.800061 17422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:36:50.800216 17422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:36:50.802155 17422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/minicluster-data/master-0/data/instance:
uuid: "548a981450ee4b91a8b17322e8356470"
format_stamp: "Formatted at 2026-09-25 16:36:50 on dist-test-slave-65pf"
server_key: "e26a57a3a9906f1227f8b96abb82c265"
server_key_iv: "543b353212b6d65a810c9b436278dc10"
server_key_version: "encryptionkey@0"
I20260925 16:36:50.802717 17422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/minicluster-data/master-0/wal/instance:
uuid: "548a981450ee4b91a8b17322e8356470"
format_stamp: "Formatted at 2026-09-25 16:36:50 on dist-test-slave-65pf"
server_key: "e26a57a3a9906f1227f8b96abb82c265"
server_key_iv: "543b353212b6d65a810c9b436278dc10"
server_key_version: "encryptionkey@0"
I20260925 16:36:50.806157 17422 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 16:36:50.808518 17437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:50.809687 17422 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:36:50.809808 17422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/minicluster-data/master-0/wal
uuid: "548a981450ee4b91a8b17322e8356470"
format_stamp: "Formatted at 2026-09-25 16:36:50 on dist-test-slave-65pf"
server_key: "e26a57a3a9906f1227f8b96abb82c265"
server_key_iv: "543b353212b6d65a810c9b436278dc10"
server_key_version: "encryptionkey@0"
I20260925 16:36:50.809913 17422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:36:50.824132 17422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:36:50.826942 17422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:36:50.827136 17422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:36:50.833998 17422 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33759
I20260925 16:36:50.834013 17489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33759 every 8 connection(s)
I20260925 16:36:50.835002 17422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:36:50.837700 17490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:36:50.841605 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 17422
I20260925 16:36:50.841754 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/minicluster-data/master-0/wal/instance
I20260925 16:36:50.842038 392 external_mini_cluster.cc:1428] Setting key c8407d8983ba45380dd2934091a8e84f
I20260925 16:36:50.843585 17490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 548a981450ee4b91a8b17322e8356470: Bootstrap starting.
I20260925 16:36:50.846037 17490 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 548a981450ee4b91a8b17322e8356470: Neither blocks nor log segments found. Creating new log.
I20260925 16:36:50.846829 17490 log.cc:824] T 00000000000000000000000000000000 P 548a981450ee4b91a8b17322e8356470: Log is configured to *not* fsync() on all Append() calls
I20260925 16:36:50.848810 17490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 548a981450ee4b91a8b17322e8356470: No bootstrap required, opened a new log
Sep 25 16:36:50 dist-test-slave-65pf krb5kdc[17409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354210, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:36:50.853300 17490 raft_consensus.cc:348] T 00000000000000000000000000000000 P 548a981450ee4b91a8b17322e8356470 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "548a981450ee4b91a8b17322e8356470" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33759 } }
I20260925 16:36:50.853577 17490 raft_consensus.cc:374] T 00000000000000000000000000000000 P 548a981450ee4b91a8b17322e8356470 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:36:50.853643 17490 raft_consensus.cc:729] T 00000000000000000000000000000000 P 548a981450ee4b91a8b17322e8356470 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548a981450ee4b91a8b17322e8356470, State: Initialized, Role: FOLLOWER
I20260925 16:36:50.854043 17490 consensus_queue.cc:260] T 00000000000000000000000000000000 P 548a981450ee4b91a8b17322e8356470 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "548a981450ee4b91a8b17322e8356470" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33759 } }
I20260925 16:36:50.854187 17490 raft_consensus.cc:388] T 00000000000000000000000000000000 P 548a981450ee4b91a8b17322e8356470 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:36:50.854269 17490 raft_consensus.cc:482] T 00000000000000000000000000000000 P 548a981450ee4b91a8b17322e8356470 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:36:50.854377 17490 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 548a981450ee4b91a8b17322e8356470 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:36:50.855242 17490 raft_consensus.cc:504] T 00000000000000000000000000000000 P 548a981450ee4b91a8b17322e8356470 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "548a981450ee4b91a8b17322e8356470" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33759 } }
I20260925 16:36:50.855600 17490 leader_election.cc:302] T 00000000000000000000000000000000 P 548a981450ee4b91a8b17322e8356470 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 548a981450ee4b91a8b17322e8356470; no voters:
I20260925 16:36:50.855942 17490 leader_election.cc:288] T 00000000000000000000000000000000 P 548a981450ee4b91a8b17322e8356470 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:36:50.856033 17495 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 548a981450ee4b91a8b17322e8356470 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:36:50.856215 17495 raft_consensus.cc:686] T 00000000000000000000000000000000 P 548a981450ee4b91a8b17322e8356470 [term 1 LEADER]: Becoming Leader. State: Replica: 548a981450ee4b91a8b17322e8356470, State: Running, Role: LEADER
I20260925 16:36:50.856410 17493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:50.843350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42316 (local address 127.0.98.62:33759)
0925 16:36:50.843740 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:50.843755 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:50.843789 (+ 34us) server_negotiation.cc:407] Connection header received
0925 16:36:50.844426 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:50.844441 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:50.844752 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:50.845023 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:36:50.845907 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:50.847033 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:50.848124 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:50.848569 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:50.848688 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:50.851364 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:36:50.851372 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:36:50.851387 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:36:50.851428 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:36:50.853289 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:36:50.853736 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:36:50.853743 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:36:50.853749 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:36:50.853901 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:36:50.854178 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:36:50.854184 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:36:50.854186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:36:50.854718 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:36:50.854923 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:50.855173 (+ 250us) server_negotiation.cc:299] Negotiation successful
0925 16:36:50.855454 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":129,"threads_started":1}
I20260925 16:36:50.856555 17495 consensus_queue.cc:237] T 00000000000000000000000000000000 P 548a981450ee4b91a8b17322e8356470 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "548a981450ee4b91a8b17322e8356470" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33759 } }
I20260925 16:36:50.857209 17490 sys_catalog.cc:564] T 00000000000000000000000000000000 P 548a981450ee4b91a8b17322e8356470 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:36:50.857720 17497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 548a981450ee4b91a8b17322e8356470 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 548a981450ee4b91a8b17322e8356470. Latest consensus state: current_term: 1 leader_uuid: "548a981450ee4b91a8b17322e8356470" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "548a981450ee4b91a8b17322e8356470" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33759 } } }
I20260925 16:36:50.858484 17497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 548a981450ee4b91a8b17322e8356470 [sys.catalog]: This master's current role is: LEADER
I20260925 16:36:50.858368 17496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 548a981450ee4b91a8b17322e8356470 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "548a981450ee4b91a8b17322e8356470" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "548a981450ee4b91a8b17322e8356470" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33759 } } }
I20260925 16:36:50.858863 17496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 548a981450ee4b91a8b17322e8356470 [sys.catalog]: This master's current role is: LEADER
I20260925 16:36:50.859634 17504 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:36:50.862159 17504 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:36:50.867565 17504 catalog_manager.cc:1347] Generated new cluster ID: 86742ed6a6c84590b30bdc2568b0ea7c
I20260925 16:36:50.867633 17504 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:36:50.895324 17504 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:36:50.896646 17504 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:36:50.901314 17504 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 548a981450ee4b91a8b17322e8356470: Generated new TSK 0
I20260925 16:36:50.902112 17504 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 16:36:50.937358 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33759
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:36:51.046675 17516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:36:51.047029 17516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:36:51.047124 17516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:36:51.051088 17516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:36:51.051172 17516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:36:51.051271 17516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 16:36:51.056471 17516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:36:51.057564 17516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:36:51.058629 17516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:36:51.067880 17522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:36:51.068015 17521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:36:51.068015 17524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:36:51.068321 17516 server_base.cc:1034] running on GCE node
I20260925 16:36:51.068749 17516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:36:51.069367 17516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:36:51.070557 17516 hybrid_clock.cc:648] HybridClock initialized: now 1790354211070541 us; error 28 us; skew 500 ppm
Sep 25 16:36:51 dist-test-slave-65pf krb5kdc[17409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354211, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:36:51.073138 17516 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 16:36:51.073958 17516 webserver.cc:460] Webserver started at http://127.0.98.1:33269/ using document root <none> and password file <none>
I20260925 16:36:51.074510 17516 fs_manager.cc:359] Metadata directory not provided
I20260925 16:36:51.074617 17516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:36:51.074847 17516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:36:51.076834 17516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/minicluster-data/ts-0/data/instance:
uuid: "32654602b42b400cbc853b4422483634"
format_stamp: "Formatted at 2026-09-25 16:36:51 on dist-test-slave-65pf"
server_key: "e51bb51b0e42a688ff82410ff021f345"
server_key_iv: "f62ea39b5f4cbbcd35d3ab4e99d3b4ed"
server_key_version: "encryptionkey@0"
I20260925 16:36:51.077379 17516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/minicluster-data/ts-0/wal/instance:
uuid: "32654602b42b400cbc853b4422483634"
format_stamp: "Formatted at 2026-09-25 16:36:51 on dist-test-slave-65pf"
server_key: "e51bb51b0e42a688ff82410ff021f345"
server_key_iv: "f62ea39b5f4cbbcd35d3ab4e99d3b4ed"
server_key_version: "encryptionkey@0"
I20260925 16:36:51.080992 17516 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260925 16:36:51.083393 17531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:51.084544 17516 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260925 16:36:51.084671 17516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/minicluster-data/ts-0/wal
uuid: "32654602b42b400cbc853b4422483634"
format_stamp: "Formatted at 2026-09-25 16:36:51 on dist-test-slave-65pf"
server_key: "e51bb51b0e42a688ff82410ff021f345"
server_key_iv: "f62ea39b5f4cbbcd35d3ab4e99d3b4ed"
server_key_version: "encryptionkey@0"
I20260925 16:36:51.084777 17516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:36:51.103013 17516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:36:51.105856 17516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:36:51.106060 17516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:36:51.106658 17516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:36:51.107463 17516 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:36:51.107530 17516 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:51.107600 17516 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:36:51.107652 17516 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:51.116369 17516 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39077
I20260925 16:36:51.116398 17644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39077 every 8 connection(s)
I20260925 16:36:51.117228 17516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
I20260925 16:36:51.124325 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 17516
I20260925 16:36:51.124495 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/minicluster-data/ts-0/wal/instance
I20260925 16:36:51.124787 392 external_mini_cluster.cc:1428] Setting key cf319f3124688ca2d5a86b25da0bd96f
Sep 25 16:36:51 dist-test-slave-65pf krb5kdc[17409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354211, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:36:51.131318 17493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:51.119044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34081 (local address 127.0.98.62:33759)
0925 16:36:51.119205 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:51.119210 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:51.120014 (+ 804us) server_negotiation.cc:407] Connection header received
0925 16:36:51.121257 (+ 1243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:51.121260 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:51.121309 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:51.121385 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:36:51.122506 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:51.123050 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:51.124196 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:51.124407 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:51.124536 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:51.127247 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:36:51.127254 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:36:51.127261 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:36:51.127313 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:36:51.128967 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:36:51.129496 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:36:51.129503 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:36:51.129508 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:36:51.129601 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:36:51.129914 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:36:51.129920 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:36:51.129925 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:36:51.130180 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:36:51.130288 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:51.130926 (+ 638us) server_negotiation.cc:299] Negotiation successful
0925 16:36:51.131078 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 16:36:51.132038 17647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:51.119437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33759 (local address 127.0.98.1:34081)
0925 16:36:51.119863 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:51.119909 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 16:36:51.121066 (+ 1157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:36:51.121555 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:36:51.121561 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:36:51.121898 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:36:51.122366 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:51.122380 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:51.123154 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:51.123156 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:51.124076 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:51.124090 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:51.124607 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:51.124612 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:51.124775 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:51.125407 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:36:51.125445 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:36:51.127025 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:36:51.129089 (+ 2064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:36:51.129094 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:36:51.129106 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:36:51.129396 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:36:51.129693 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:36:51.129694 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:36:51.129696 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:36:51.129823 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:36:51.130281 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:36:51.130286 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:36:51.130639 (+ 353us) client_negotiation.cc:769] Sending connection context
0925 16:36:51.130838 (+ 199us) client_negotiation.cc:240] Negotiation successful
0925 16:36:51.131123 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":114,"threads_started":1}
I20260925 16:36:51.133301 17645 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33759
I20260925 16:36:51.133580 17645 heartbeater.cc:464] Registering TS with master...
I20260925 16:36:51.134188 17645 heartbeater.cc:511] Master 127.0.98.62:33759 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 16:36:51.135844 17454 ts_manager.cc:194] Registered new tserver with Master: 32654602b42b400cbc853b4422483634 (127.0.98.1:39077)
I20260925 16:36:51.137059 17454 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34081
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 16:36:51.153604 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33759
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:36:51.264329 17650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:36:51.264580 17650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:36:51.264624 17650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:36:51.268553 17650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:36:51.268620 17650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:36:51.268678 17650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 16:36:51.273952 17650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:36:51.275036 17650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:36:51.276001 17650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:36:51.285498 17655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:36:51.285576 17650 server_base.cc:1034] running on GCE node
W20260925 16:36:51.285501 17658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:36:51.285495 17656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:36:51.286144 17650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:36:51.286832 17650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:36:51.287999 17650 hybrid_clock.cc:648] HybridClock initialized: now 1790354211287984 us; error 27 us; skew 500 ppm
Sep 25 16:36:51 dist-test-slave-65pf krb5kdc[17409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354211, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:36:51.290805 17650 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 16:36:51.291630 17650 webserver.cc:460] Webserver started at http://127.0.98.2:46141/ using document root <none> and password file <none>
I20260925 16:36:51.292199 17650 fs_manager.cc:359] Metadata directory not provided
I20260925 16:36:51.292287 17650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:36:51.292520 17650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:36:51.294533 17650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/minicluster-data/ts-1/data/instance:
uuid: "115e268a2ec4445e9279b99a26c67c15"
format_stamp: "Formatted at 2026-09-25 16:36:51 on dist-test-slave-65pf"
server_key: "270de4e5bb19c598b999393ad6454dea"
server_key_iv: "c226fafc4d2645b3ee96debab5bee995"
server_key_version: "encryptionkey@0"
I20260925 16:36:51.295151 17650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/minicluster-data/ts-1/wal/instance:
uuid: "115e268a2ec4445e9279b99a26c67c15"
format_stamp: "Formatted at 2026-09-25 16:36:51 on dist-test-slave-65pf"
server_key: "270de4e5bb19c598b999393ad6454dea"
server_key_iv: "c226fafc4d2645b3ee96debab5bee995"
server_key_version: "encryptionkey@0"
I20260925 16:36:51.298846 17650 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 16:36:51.301177 17665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:51.302351 17650 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:36:51.302476 17650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/minicluster-data/ts-1/wal
uuid: "115e268a2ec4445e9279b99a26c67c15"
format_stamp: "Formatted at 2026-09-25 16:36:51 on dist-test-slave-65pf"
server_key: "270de4e5bb19c598b999393ad6454dea"
server_key_iv: "c226fafc4d2645b3ee96debab5bee995"
server_key_version: "encryptionkey@0"
I20260925 16:36:51.302580 17650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:36:51.316682 17650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:36:51.320109 17650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:36:51.320308 17650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:36:51.320890 17650 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:36:51.321691 17650 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:36:51.321760 17650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:51.321831 17650 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:36:51.321889 17650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:51.331019 17650 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33139
I20260925 16:36:51.331243 17778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33139 every 8 connection(s)
I20260925 16:36:51.331947 17650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
I20260925 16:36:51.340152 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 17650
I20260925 16:36:51.340329 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/minicluster-data/ts-1/wal/instance
I20260925 16:36:51.340637 392 external_mini_cluster.cc:1428] Setting key 0d27cecf9133efb293b31310fc6f67c0
Sep 25 16:36:51 dist-test-slave-65pf krb5kdc[17409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354211, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:36:51.346154 17493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:51.333717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35439 (local address 127.0.98.62:33759)
0925 16:36:51.333848 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:51.333852 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:51.334716 (+ 864us) server_negotiation.cc:407] Connection header received
0925 16:36:51.336044 (+ 1328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:51.336048 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:51.336104 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:51.336215 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:36:51.337369 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:51.337896 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:51.339057 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:51.339293 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:51.339403 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:51.342217 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:36:51.342220 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:36:51.342222 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:36:51.342245 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:36:51.343977 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:36:51.344462 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:36:51.344467 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:36:51.344469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:36:51.344571 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:36:51.344915 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:36:51.344920 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:36:51.344925 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:36:51.345179 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:36:51.345283 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:51.345780 (+ 497us) server_negotiation.cc:299] Negotiation successful
0925 16:36:51.345935 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 16:36:51.347007 17781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:51.334089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33759 (local address 127.0.98.2:35439)
0925 16:36:51.334522 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:51.334562 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 16:36:51.335829 (+ 1267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:36:51.336398 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:36:51.336405 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:36:51.336761 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:36:51.337221 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:51.337234 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:51.338021 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:51.338025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:51.338931 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:51.338939 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:51.339441 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:51.339445 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:51.339612 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:51.340352 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:36:51.340392 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:36:51.342007 (+ 1615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:36:51.344097 (+ 2090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:36:51.344102 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:36:51.344114 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:36:51.344365 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:36:51.344663 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:36:51.344665 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:36:51.344666 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:36:51.344820 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:36:51.345290 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:36:51.345295 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:36:51.345542 (+ 247us) client_negotiation.cc:769] Sending connection context
0925 16:36:51.345698 (+ 156us) client_negotiation.cc:240] Negotiation successful
0925 16:36:51.345982 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":113,"threads_started":1}
I20260925 16:36:51.348251 17779 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33759
I20260925 16:36:51.348487 17779 heartbeater.cc:464] Registering TS with master...
I20260925 16:36:51.348987 17779 heartbeater.cc:511] Master 127.0.98.62:33759 requested a full tablet report, sending...
I20260925 16:36:51.350032 17454 ts_manager.cc:194] Registered new tserver with Master: 115e268a2ec4445e9279b99a26c67c15 (127.0.98.2:33139)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260925 16:36:51.350687 17454 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35439
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 16:36:51.369894 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33759
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:36:51.478618 17784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:36:51.478931 17784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:36:51.479009 17784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:36:51.482997 17784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:36:51.483088 17784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:36:51.483181 17784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 16:36:51.488425 17784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:36:51.489518 17784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:36:51.490520 17784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:36:51.499617 17789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:36:51.499614 17792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:36:51.499750 17790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:36:51.499998 17784 server_base.cc:1034] running on GCE node
I20260925 16:36:51.500370 17784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:36:51.500989 17784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:36:51.502164 17784 hybrid_clock.cc:648] HybridClock initialized: now 1790354211502132 us; error 51 us; skew 500 ppm
Sep 25 16:36:51 dist-test-slave-65pf krb5kdc[17409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354211, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:36:51.504848 17784 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 16:36:51.505746 17784 webserver.cc:460] Webserver started at http://127.0.98.3:40623/ using document root <none> and password file <none>
I20260925 16:36:51.506327 17784 fs_manager.cc:359] Metadata directory not provided
I20260925 16:36:51.506420 17784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:36:51.506712 17784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:36:51.508658 17784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/minicluster-data/ts-2/data/instance:
uuid: "bb988f82fff54b6e8d468bd6075898b2"
format_stamp: "Formatted at 2026-09-25 16:36:51 on dist-test-slave-65pf"
server_key: "6a2c59e9771cca33845bb8c64d61c298"
server_key_iv: "e9d9f346e0dc6e4784e11e210e81c982"
server_key_version: "encryptionkey@0"
I20260925 16:36:51.509228 17784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/minicluster-data/ts-2/wal/instance:
uuid: "bb988f82fff54b6e8d468bd6075898b2"
format_stamp: "Formatted at 2026-09-25 16:36:51 on dist-test-slave-65pf"
server_key: "6a2c59e9771cca33845bb8c64d61c298"
server_key_iv: "e9d9f346e0dc6e4784e11e210e81c982"
server_key_version: "encryptionkey@0"
I20260925 16:36:51.512739 17784 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 16:36:51.515275 17799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:51.516516 17784 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:36:51.516644 17784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/minicluster-data/ts-2/wal
uuid: "bb988f82fff54b6e8d468bd6075898b2"
format_stamp: "Formatted at 2026-09-25 16:36:51 on dist-test-slave-65pf"
server_key: "6a2c59e9771cca33845bb8c64d61c298"
server_key_iv: "e9d9f346e0dc6e4784e11e210e81c982"
server_key_version: "encryptionkey@0"
I20260925 16:36:51.516747 17784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:36:51.527500 17784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:36:51.530727 17784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:36:51.530921 17784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:36:51.531484 17784 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:36:51.532272 17784 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:36:51.532356 17784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:51.532449 17784 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:36:51.532500 17784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:51.541608 17784 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44505
I20260925 16:36:51.541635 17912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44505 every 8 connection(s)
I20260925 16:36:51.542464 17784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
I20260925 16:36:51.546060 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 17784
I20260925 16:36:51.546217 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/minicluster-data/ts-2/wal/instance
I20260925 16:36:51.546569 392 external_mini_cluster.cc:1428] Setting key 400673c35d36e019ae7192ec674be8b2
Sep 25 16:36:51 dist-test-slave-65pf krb5kdc[17409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354211, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:36:51.555168 17493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:51.544262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44399 (local address 127.0.98.62:33759)
0925 16:36:51.544397 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:51.544401 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:51.545183 (+ 782us) server_negotiation.cc:407] Connection header received
0925 16:36:51.546496 (+ 1313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:51.546499 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:51.546558 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:51.546694 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:36:51.547889 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:51.548492 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:51.549617 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:51.549828 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:51.549902 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:51.552132 (+ 2230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:36:51.552135 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:36:51.552137 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:36:51.552157 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:36:51.553509 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:36:51.553918 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:36:51.553920 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:36:51.553921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:36:51.553987 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:36:51.554257 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:36:51.554259 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:36:51.554260 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:36:51.554415 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:36:51.554484 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:51.554971 (+ 487us) server_negotiation.cc:299] Negotiation successful
0925 16:36:51.555058 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 16:36:51.555953 17915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:51.544631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33759 (local address 127.0.98.3:44399)
0925 16:36:51.545039 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:51.545078 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 16:36:51.546261 (+ 1183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:36:51.546881 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:36:51.546888 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:36:51.547270 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:36:51.547786 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:51.547799 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:51.548618 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:51.548621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:51.549487 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:51.549496 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:51.549946 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:51.549948 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:51.550081 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:51.550607 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:36:51.550622 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:36:51.551998 (+ 1376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:36:51.553602 (+ 1604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:36:51.553606 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:36:51.553615 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:36:51.553838 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:36:51.554059 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:36:51.554061 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:36:51.554062 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:36:51.554176 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:36:51.554494 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:36:51.554499 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:36:51.554776 (+ 277us) client_negotiation.cc:769] Sending connection context
0925 16:36:51.554926 (+ 150us) client_negotiation.cc:240] Negotiation successful
0925 16:36:51.555148 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":109,"threads_started":1}
I20260925 16:36:51.556990 17913 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33759
I20260925 16:36:51.557214 17913 heartbeater.cc:464] Registering TS with master...
I20260925 16:36:51.557819 17913 heartbeater.cc:511] Master 127.0.98.62:33759 requested a full tablet report, sending...
I20260925 16:36:51.558912 17454 ts_manager.cc:194] Registered new tserver with Master: bb988f82fff54b6e8d468bd6075898b2 (127.0.98.3:44505)
I20260925 16:36:51.559461 17454 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44399
I20260925 16:36:51.560293 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 16:36:51.568986 17493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:51.562382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42318 (local address 127.0.98.62:33759)
0925 16:36:51.562490 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:51.562493 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:51.562685 (+ 192us) server_negotiation.cc:407] Connection header received
0925 16:36:51.562783 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:51.562785 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:51.562819 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:51.562882 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:36:51.563519 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:51.563994 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:51.564792 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:51.564990 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:51.565064 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:51.566319 (+ 1255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:36:51.566322 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:36:51.566324 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:36:51.566343 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:36:51.567566 (+ 1223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:36:51.567913 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:36:51.567915 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:36:51.567916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:36:51.567982 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:36:51.568239 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:36:51.568242 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:36:51.568243 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:36:51.568466 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:36:51.568599 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:51.568759 (+ 160us) server_negotiation.cc:299] Negotiation successful
0925 16:36:51.568855 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 16:36:51.571731 17454 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42318:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 16:36:51.574555 17454 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:36:51.588248 17924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:51.582699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39077 (local address 127.0.0.1:60246)
0925 16:36:51.583192 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:51.583237 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 16:36:51.583369 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:36:51.584141 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:36:51.584147 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:36:51.584171 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:36:51.584293 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:51.584300 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:51.585619 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:51.585622 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:51.586481 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:51.586494 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:51.587717 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:51.587720 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:51.587857 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:51.587895 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 16:36:51.587997 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 16:36:51.588100 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":1,"thread_start_us":77,"threads_started":1}
I20260925 16:36:51.588399 17925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:51.582900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60246 (local address 127.0.98.1:39077)
0925 16:36:51.583256 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:51.583274 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:51.583300 (+ 26us) server_negotiation.cc:407] Connection header received
0925 16:36:51.583587 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:51.583595 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:51.583796 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:51.583957 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:36:51.584448 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:51.585490 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:51.586748 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:51.587601 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:51.587713 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:51.587873 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:51.588050 (+ 177us) server_negotiation.cc:299] Negotiation successful
0925 16:36:51.588144 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":87,"threads_started":1}
I20260925 16:36:51.589089 17927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:51.583182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44505 (local address 127.0.0.1:49960)
0925 16:36:51.583895 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:51.583910 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:36:51.584006 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:36:51.585033 (+ 1027us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:36:51.585036 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:36:51.585054 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:36:51.585137 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:51.585142 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:51.586526 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:51.586532 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:51.587562 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:51.587571 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:51.588680 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:51.588687 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:51.588784 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:51.588819 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 16:36:51.588869 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 16:36:51.588939 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":113,"threads_started":1}
I20260925 16:36:51.589396 17926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:51.583440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33139 (local address 127.0.0.1:38288)
0925 16:36:51.584211 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:51.584224 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:36:51.584326 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:36:51.584859 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:36:51.584862 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:36:51.584879 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:36:51.585006 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:51.585016 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:51.586571 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:51.586575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:51.587800 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:51.587810 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:51.589039 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:51.589043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:51.589136 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:51.589163 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 16:36:51.589226 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 16:36:51.589286 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":715,"mutex_wait_us":26,"thread_start_us":84,"threads_started":1}
I20260925 16:36:51.589700 17928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:51.583619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38288 (local address 127.0.98.2:33139)
0925 16:36:51.584233 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:51.584246 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:51.584294 (+ 48us) server_negotiation.cc:407] Connection header received
0925 16:36:51.584551 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:51.584556 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:51.584718 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:51.584862 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:36:51.585149 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:51.586378 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:51.587940 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:51.588913 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:51.589038 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:51.589372 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:51.589428 (+ 56us) server_negotiation.cc:299] Negotiation successful
0925 16:36:51.589534 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":107,"threads_started":1}
I20260925 16:36:51.589746 17929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:51.583706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49960 (local address 127.0.98.3:44505)
0925 16:36:51.584622 (+ 916us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:51.584633 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:51.584652 (+ 19us) server_negotiation.cc:407] Connection header received
0925 16:36:51.584723 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:51.584727 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:51.584904 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:51.585059 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:36:51.585242 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:51.586376 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:51.587685 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:51.588551 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:51.589248 (+ 697us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:51.589398 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:51.589449 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 16:36:51.589557 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":807,"thread_start_us":149,"threads_started":1}
I20260925 16:36:51.591763 17579 tablet_service.cc:1467] Processing CreateTablet for tablet c47b5fc45d094a6c92f8573a8c59c5fd (DEFAULT_TABLE table=test-table [id=33d9177dfa5d4faf84482801c104b13a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:36:51.592762 17579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c47b5fc45d094a6c92f8573a8c59c5fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:36:51.592386 17713 tablet_service.cc:1467] Processing CreateTablet for tablet c47b5fc45d094a6c92f8573a8c59c5fd (DEFAULT_TABLE table=test-table [id=33d9177dfa5d4faf84482801c104b13a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:36:51.592849 17847 tablet_service.cc:1467] Processing CreateTablet for tablet c47b5fc45d094a6c92f8573a8c59c5fd (DEFAULT_TABLE table=test-table [id=33d9177dfa5d4faf84482801c104b13a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:36:51.593358 17713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c47b5fc45d094a6c92f8573a8c59c5fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:36:51.593787 17847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c47b5fc45d094a6c92f8573a8c59c5fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:36:51.600025 17931 tablet_bootstrap.cc:492] T c47b5fc45d094a6c92f8573a8c59c5fd P 115e268a2ec4445e9279b99a26c67c15: Bootstrap starting.
I20260925 16:36:51.600091 17930 tablet_bootstrap.cc:492] T c47b5fc45d094a6c92f8573a8c59c5fd P 32654602b42b400cbc853b4422483634: Bootstrap starting.
I20260925 16:36:51.600461 17932 tablet_bootstrap.cc:492] T c47b5fc45d094a6c92f8573a8c59c5fd P bb988f82fff54b6e8d468bd6075898b2: Bootstrap starting.
I20260925 16:36:51.602936 17932 tablet_bootstrap.cc:654] T c47b5fc45d094a6c92f8573a8c59c5fd P bb988f82fff54b6e8d468bd6075898b2: Neither blocks nor log segments found. Creating new log.
I20260925 16:36:51.602936 17930 tablet_bootstrap.cc:654] T c47b5fc45d094a6c92f8573a8c59c5fd P 32654602b42b400cbc853b4422483634: Neither blocks nor log segments found. Creating new log.
I20260925 16:36:51.604206 17931 tablet_bootstrap.cc:654] T c47b5fc45d094a6c92f8573a8c59c5fd P 115e268a2ec4445e9279b99a26c67c15: Neither blocks nor log segments found. Creating new log.
I20260925 16:36:51.604257 17932 log.cc:824] T c47b5fc45d094a6c92f8573a8c59c5fd P bb988f82fff54b6e8d468bd6075898b2: Log is configured to *not* fsync() on all Append() calls
I20260925 16:36:51.604257 17930 log.cc:824] T c47b5fc45d094a6c92f8573a8c59c5fd P 32654602b42b400cbc853b4422483634: Log is configured to *not* fsync() on all Append() calls
I20260925 16:36:51.605391 17931 log.cc:824] T c47b5fc45d094a6c92f8573a8c59c5fd P 115e268a2ec4445e9279b99a26c67c15: Log is configured to *not* fsync() on all Append() calls
I20260925 16:36:51.606413 17930 tablet_bootstrap.cc:492] T c47b5fc45d094a6c92f8573a8c59c5fd P 32654602b42b400cbc853b4422483634: No bootstrap required, opened a new log
I20260925 16:36:51.606626 17930 ts_tablet_manager.cc:1397] T c47b5fc45d094a6c92f8573a8c59c5fd P 32654602b42b400cbc853b4422483634: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260925 16:36:51.606706 17932 tablet_bootstrap.cc:492] T c47b5fc45d094a6c92f8573a8c59c5fd P bb988f82fff54b6e8d468bd6075898b2: No bootstrap required, opened a new log
I20260925 16:36:51.606894 17932 ts_tablet_manager.cc:1397] T c47b5fc45d094a6c92f8573a8c59c5fd P bb988f82fff54b6e8d468bd6075898b2: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260925 16:36:51.607252 17931 tablet_bootstrap.cc:492] T c47b5fc45d094a6c92f8573a8c59c5fd P 115e268a2ec4445e9279b99a26c67c15: No bootstrap required, opened a new log
I20260925 16:36:51.607414 17931 ts_tablet_manager.cc:1397] T c47b5fc45d094a6c92f8573a8c59c5fd P 115e268a2ec4445e9279b99a26c67c15: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260925 16:36:51.609817 17932 raft_consensus.cc:348] T c47b5fc45d094a6c92f8573a8c59c5fd P bb988f82fff54b6e8d468bd6075898b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32654602b42b400cbc853b4422483634" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39077 } } peers { permanent_uuid: "115e268a2ec4445e9279b99a26c67c15" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33139 } } peers { permanent_uuid: "bb988f82fff54b6e8d468bd6075898b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44505 } }
I20260925 16:36:51.610078 17932 raft_consensus.cc:374] T c47b5fc45d094a6c92f8573a8c59c5fd P bb988f82fff54b6e8d468bd6075898b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:36:51.610157 17932 raft_consensus.cc:729] T c47b5fc45d094a6c92f8573a8c59c5fd P bb988f82fff54b6e8d468bd6075898b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb988f82fff54b6e8d468bd6075898b2, State: Initialized, Role: FOLLOWER
I20260925 16:36:51.610473 17930 raft_consensus.cc:348] T c47b5fc45d094a6c92f8573a8c59c5fd P 32654602b42b400cbc853b4422483634 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32654602b42b400cbc853b4422483634" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39077 } } peers { permanent_uuid: "115e268a2ec4445e9279b99a26c67c15" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33139 } } peers { permanent_uuid: "bb988f82fff54b6e8d468bd6075898b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44505 } }
I20260925 16:36:51.610747 17930 raft_consensus.cc:374] T c47b5fc45d094a6c92f8573a8c59c5fd P 32654602b42b400cbc853b4422483634 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:36:51.610677 17932 consensus_queue.cc:260] T c47b5fc45d094a6c92f8573a8c59c5fd P bb988f82fff54b6e8d468bd6075898b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32654602b42b400cbc853b4422483634" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39077 } } peers { permanent_uuid: "115e268a2ec4445e9279b99a26c67c15" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33139 } } peers { permanent_uuid: "bb988f82fff54b6e8d468bd6075898b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44505 } }
I20260925 16:36:51.610842 17930 raft_consensus.cc:729] T c47b5fc45d094a6c92f8573a8c59c5fd P 32654602b42b400cbc853b4422483634 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32654602b42b400cbc853b4422483634, State: Initialized, Role: FOLLOWER
I20260925 16:36:51.611027 17931 raft_consensus.cc:348] T c47b5fc45d094a6c92f8573a8c59c5fd P 115e268a2ec4445e9279b99a26c67c15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32654602b42b400cbc853b4422483634" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39077 } } peers { permanent_uuid: "115e268a2ec4445e9279b99a26c67c15" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33139 } } peers { permanent_uuid: "bb988f82fff54b6e8d468bd6075898b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44505 } }
I20260925 16:36:51.611305 17931 raft_consensus.cc:374] T c47b5fc45d094a6c92f8573a8c59c5fd P 115e268a2ec4445e9279b99a26c67c15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:36:51.611299 17930 consensus_queue.cc:260] T c47b5fc45d094a6c92f8573a8c59c5fd P 32654602b42b400cbc853b4422483634 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32654602b42b400cbc853b4422483634" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39077 } } peers { permanent_uuid: "115e268a2ec4445e9279b99a26c67c15" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33139 } } peers { permanent_uuid: "bb988f82fff54b6e8d468bd6075898b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44505 } }
I20260925 16:36:51.611397 17931 raft_consensus.cc:729] T c47b5fc45d094a6c92f8573a8c59c5fd P 115e268a2ec4445e9279b99a26c67c15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 115e268a2ec4445e9279b99a26c67c15, State: Initialized, Role: FOLLOWER
I20260925 16:36:51.611410 17913 heartbeater.cc:503] Master 127.0.98.62:33759 was elected leader, sending a full tablet report...
I20260925 16:36:51.611833 17931 consensus_queue.cc:260] T c47b5fc45d094a6c92f8573a8c59c5fd P 115e268a2ec4445e9279b99a26c67c15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32654602b42b400cbc853b4422483634" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39077 } } peers { permanent_uuid: "115e268a2ec4445e9279b99a26c67c15" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33139 } } peers { permanent_uuid: "bb988f82fff54b6e8d468bd6075898b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44505 } }
I20260925 16:36:51.612242 17932 ts_tablet_manager.cc:1428] T c47b5fc45d094a6c92f8573a8c59c5fd P bb988f82fff54b6e8d468bd6075898b2: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 16:36:51.612370 17930 ts_tablet_manager.cc:1428] T c47b5fc45d094a6c92f8573a8c59c5fd P 32654602b42b400cbc853b4422483634: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260925 16:36:51.612677 17645 heartbeater.cc:503] Master 127.0.98.62:33759 was elected leader, sending a full tablet report...
I20260925 16:36:51.612741 17931 ts_tablet_manager.cc:1428] T c47b5fc45d094a6c92f8573a8c59c5fd P 115e268a2ec4445e9279b99a26c67c15: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 16:36:51.612819 17779 heartbeater.cc:503] Master 127.0.98.62:33759 was elected leader, sending a full tablet report...
W20260925 16:36:51.618356 17646 tablet.cc:2369] T c47b5fc45d094a6c92f8573a8c59c5fd P 32654602b42b400cbc853b4422483634: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 16:36:51.793522 17914 tablet.cc:2369] T c47b5fc45d094a6c92f8573a8c59c5fd P bb988f82fff54b6e8d468bd6075898b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 16:36:51.833132 17780 tablet.cc:2369] T c47b5fc45d094a6c92f8573a8c59c5fd P 115e268a2ec4445e9279b99a26c67c15: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:36:51.840586 17938 raft_consensus.cc:482] T c47b5fc45d094a6c92f8573a8c59c5fd P 115e268a2ec4445e9279b99a26c67c15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:36:51.840749 17938 raft_consensus.cc:504] T c47b5fc45d094a6c92f8573a8c59c5fd P 115e268a2ec4445e9279b99a26c67c15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32654602b42b400cbc853b4422483634" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39077 } } peers { permanent_uuid: "115e268a2ec4445e9279b99a26c67c15" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33139 } } peers { permanent_uuid: "bb988f82fff54b6e8d468bd6075898b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44505 } }
I20260925 16:36:51.841988 17938 leader_election.cc:288] T c47b5fc45d094a6c92f8573a8c59c5fd P 115e268a2ec4445e9279b99a26c67c15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 32654602b42b400cbc853b4422483634 (127.0.98.1:39077), bb988f82fff54b6e8d468bd6075898b2 (127.0.98.3:44505)
I20260925 16:36:51.846212 17925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:51.842722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41871 (local address 127.0.98.1:39077)
0925 16:36:51.842852 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:51.842857 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:51.842869 (+ 12us) server_negotiation.cc:407] Connection header received
0925 16:36:51.842920 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:51.842923 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:51.842969 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:51.843045 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:36:51.843401 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:51.844114 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:51.845244 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:51.845678 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:51.845772 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:51.845847 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:51.846021 (+ 174us) server_negotiation.cc:299] Negotiation successful
0925 16:36:51.846088 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 16:36:51.846235 17939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:51.842250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39077 (local address 127.0.98.2:41871)
0925 16:36:51.842634 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:51.842650 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:36:51.842762 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:36:51.843078 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:36:51.843081 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:36:51.843097 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:36:51.843271 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:51.843278 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:51.844236 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:51.844242 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:51.845110 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:51.845118 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:51.845781 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:51.845785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:51.845888 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:51.845917 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 16:36:51.845990 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 16:36:51.846063 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":69,"thread_start_us":78,"threads_started":1}
I20260925 16:36:51.846931 17781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:51.842238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44505 (local address 127.0.98.2:55197)
0925 16:36:51.842452 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:51.842467 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:36:51.842564 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:36:51.842972 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:36:51.842975 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:36:51.842994 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:36:51.843170 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:51.843176 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:51.844225 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:51.844229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:51.845571 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:51.845582 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:51.846503 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:51.846508 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:51.846652 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:51.846684 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 16:36:51.846748 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 16:36:51.846808 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":86}
I20260925 16:36:51.846884 17599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c47b5fc45d094a6c92f8573a8c59c5fd" candidate_uuid: "115e268a2ec4445e9279b99a26c67c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32654602b42b400cbc853b4422483634" is_pre_election: true
I20260925 16:36:51.847218 17599 raft_consensus.cc:2454] T c47b5fc45d094a6c92f8573a8c59c5fd P 32654602b42b400cbc853b4422483634 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 115e268a2ec4445e9279b99a26c67c15 in term 0.
I20260925 16:36:51.847280 17929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:51.842371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55197 (local address 127.0.98.3:44505)
0925 16:36:51.842497 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:51.842503 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:51.842520 (+ 17us) server_negotiation.cc:407] Connection header received
0925 16:36:51.842720 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:51.842723 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:51.842775 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:51.842852 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:36:51.843296 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:51.844097 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:51.845716 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:51.846290 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:51.846396 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:51.846449 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:51.847109 (+ 660us) server_negotiation.cc:299] Negotiation successful
0925 16:36:51.847183 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 16:36:51.847812 17666 leader_election.cc:302] T c47b5fc45d094a6c92f8573a8c59c5fd P 115e268a2ec4445e9279b99a26c67c15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 115e268a2ec4445e9279b99a26c67c15, 32654602b42b400cbc853b4422483634; no voters:
I20260925 16:36:51.847894 17864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c47b5fc45d094a6c92f8573a8c59c5fd" candidate_uuid: "115e268a2ec4445e9279b99a26c67c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb988f82fff54b6e8d468bd6075898b2" is_pre_election: true
I20260925 16:36:51.848114 17938 raft_consensus.cc:2781] T c47b5fc45d094a6c92f8573a8c59c5fd P 115e268a2ec4445e9279b99a26c67c15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:36:51.848174 17864 raft_consensus.cc:2454] T c47b5fc45d094a6c92f8573a8c59c5fd P bb988f82fff54b6e8d468bd6075898b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 115e268a2ec4445e9279b99a26c67c15 in term 0.
I20260925 16:36:51.848222 17938 raft_consensus.cc:482] T c47b5fc45d094a6c92f8573a8c59c5fd P 115e268a2ec4445e9279b99a26c67c15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:36:51.848290 17938 raft_consensus.cc:3037] T c47b5fc45d094a6c92f8573a8c59c5fd P 115e268a2ec4445e9279b99a26c67c15 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:36:51.849350 17938 raft_consensus.cc:504] T c47b5fc45d094a6c92f8573a8c59c5fd P 115e268a2ec4445e9279b99a26c67c15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32654602b42b400cbc853b4422483634" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39077 } } peers { permanent_uuid: "115e268a2ec4445e9279b99a26c67c15" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33139 } } peers { permanent_uuid: "bb988f82fff54b6e8d468bd6075898b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44505 } }
I20260925 16:36:51.849682 17938 leader_election.cc:288] T c47b5fc45d094a6c92f8573a8c59c5fd P 115e268a2ec4445e9279b99a26c67c15 [CANDIDATE]: Term 1 election: Requested vote from peers 32654602b42b400cbc853b4422483634 (127.0.98.1:39077), bb988f82fff54b6e8d468bd6075898b2 (127.0.98.3:44505)
I20260925 16:36:51.850073 17864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c47b5fc45d094a6c92f8573a8c59c5fd" candidate_uuid: "115e268a2ec4445e9279b99a26c67c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb988f82fff54b6e8d468bd6075898b2"
I20260925 16:36:51.850116 17599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c47b5fc45d094a6c92f8573a8c59c5fd" candidate_uuid: "115e268a2ec4445e9279b99a26c67c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32654602b42b400cbc853b4422483634"
I20260925 16:36:51.850201 17864 raft_consensus.cc:3037] T c47b5fc45d094a6c92f8573a8c59c5fd P bb988f82fff54b6e8d468bd6075898b2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:36:51.850216 17599 raft_consensus.cc:3037] T c47b5fc45d094a6c92f8573a8c59c5fd P 32654602b42b400cbc853b4422483634 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:36:51.851472 17864 raft_consensus.cc:2454] T c47b5fc45d094a6c92f8573a8c59c5fd P bb988f82fff54b6e8d468bd6075898b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 115e268a2ec4445e9279b99a26c67c15 in term 1.
I20260925 16:36:51.851472 17599 raft_consensus.cc:2454] T c47b5fc45d094a6c92f8573a8c59c5fd P 32654602b42b400cbc853b4422483634 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 115e268a2ec4445e9279b99a26c67c15 in term 1.
I20260925 16:36:51.851850 17666 leader_election.cc:302] T c47b5fc45d094a6c92f8573a8c59c5fd P 115e268a2ec4445e9279b99a26c67c15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 115e268a2ec4445e9279b99a26c67c15, 32654602b42b400cbc853b4422483634; no voters:
I20260925 16:36:51.852039 17938 raft_consensus.cc:2781] T c47b5fc45d094a6c92f8573a8c59c5fd P 115e268a2ec4445e9279b99a26c67c15 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:36:51.852267 17938 raft_consensus.cc:686] T c47b5fc45d094a6c92f8573a8c59c5fd P 115e268a2ec4445e9279b99a26c67c15 [term 1 LEADER]: Becoming Leader. State: Replica: 115e268a2ec4445e9279b99a26c67c15, State: Running, Role: LEADER
I20260925 16:36:51.852591 17938 consensus_queue.cc:237] T c47b5fc45d094a6c92f8573a8c59c5fd P 115e268a2ec4445e9279b99a26c67c15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32654602b42b400cbc853b4422483634" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39077 } } peers { permanent_uuid: "115e268a2ec4445e9279b99a26c67c15" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33139 } } peers { permanent_uuid: "bb988f82fff54b6e8d468bd6075898b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44505 } }
I20260925 16:36:51.858949 17454 catalog_manager.cc:5495] T c47b5fc45d094a6c92f8573a8c59c5fd P 115e268a2ec4445e9279b99a26c67c15 reported cstate change: term changed from 0 to 1, leader changed from <none> to 115e268a2ec4445e9279b99a26c67c15 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "115e268a2ec4445e9279b99a26c67c15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32654602b42b400cbc853b4422483634" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "115e268a2ec4445e9279b99a26c67c15" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb988f82fff54b6e8d468bd6075898b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44505 } health_report { overall_health: UNKNOWN } } }
I20260925 16:36:51.881768 17928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:51.877382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38292 (local address 127.0.98.2:33139)
0925 16:36:51.877503 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:51.877508 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:51.877529 (+ 21us) server_negotiation.cc:407] Connection header received
0925 16:36:51.877714 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:51.877716 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:51.877770 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:51.877863 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:36:51.878165 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:51.878747 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:51.879999 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:51.880396 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:51.880488 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:51.880959 (+ 471us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:36:51.881369 (+ 410us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:36:51.881515 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:51.881557 (+ 42us) server_negotiation.cc:299] Negotiation successful
0925 16:36:51.881607 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 16:36:51.887974 17599 raft_consensus.cc:1261] T c47b5fc45d094a6c92f8573a8c59c5fd P 32654602b42b400cbc853b4422483634 [term 1 FOLLOWER]: Refusing update from remote peer 115e268a2ec4445e9279b99a26c67c15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:36:51.888024 17864 raft_consensus.cc:1261] T c47b5fc45d094a6c92f8573a8c59c5fd P bb988f82fff54b6e8d468bd6075898b2 [term 1 FOLLOWER]: Refusing update from remote peer 115e268a2ec4445e9279b99a26c67c15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:36:51.888708 17940 consensus_queue.cc:1035] T c47b5fc45d094a6c92f8573a8c59c5fd P 115e268a2ec4445e9279b99a26c67c15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32654602b42b400cbc853b4422483634" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:36:51.888931 17938 consensus_queue.cc:1035] T c47b5fc45d094a6c92f8573a8c59c5fd P 115e268a2ec4445e9279b99a26c67c15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb988f82fff54b6e8d468bd6075898b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:36:51.896114 17943 mvcc.cc:205] Tried to move back new op lower bound from 7333290851885543424 to 7333290851751309312. Current Snapshot: MvccSnapshot[applied={T|T < 7333290851885543424}]
I20260925 16:36:51.897356 17945 mvcc.cc:205] Tried to move back new op lower bound from 7333290851885543424 to 7333290851751309312. Current Snapshot: MvccSnapshot[applied={T|T < 7333290851885543424}]
I20260925 16:36:51.905661 17452 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42318:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:36:51.905861 17452 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42318:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:36:51.908465 17452 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 548a981450ee4b91a8b17322e8356470: Sending DeleteTablet for 3 replicas of tablet c47b5fc45d094a6c92f8573a8c59c5fd
I20260925 16:36:51.909308 17847 tablet_service.cc:1514] Processing DeleteTablet for tablet c47b5fc45d094a6c92f8573a8c59c5fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:36:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49960
I20260925 16:36:51.909314 17579 tablet_service.cc:1514] Processing DeleteTablet for tablet c47b5fc45d094a6c92f8573a8c59c5fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:36:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60246
I20260925 16:36:51.909807 17953 tablet_replica.cc:330] T c47b5fc45d094a6c92f8573a8c59c5fd P bb988f82fff54b6e8d468bd6075898b2: stopping tablet replica
I20260925 16:36:51.909830 17713 tablet_service.cc:1514] Processing DeleteTablet for tablet c47b5fc45d094a6c92f8573a8c59c5fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:36:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38288
I20260925 16:36:51.910240 17954 tablet_replica.cc:330] T c47b5fc45d094a6c92f8573a8c59c5fd P 115e268a2ec4445e9279b99a26c67c15: stopping tablet replica
I20260925 16:36:51.910243 17953 raft_consensus.cc:2229] T c47b5fc45d094a6c92f8573a8c59c5fd P bb988f82fff54b6e8d468bd6075898b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:36:51.910513 17954 raft_consensus.cc:2229] T c47b5fc45d094a6c92f8573a8c59c5fd P 115e268a2ec4445e9279b99a26c67c15 [term 1 LEADER]: Raft consensus shutting down.
I20260925 16:36:51.910866 17953 raft_consensus.cc:2258] T c47b5fc45d094a6c92f8573a8c59c5fd P bb988f82fff54b6e8d468bd6075898b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:36:51.910925 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 17516
I20260925 16:36:51.910929 17954 raft_consensus.cc:2258] T c47b5fc45d094a6c92f8573a8c59c5fd P 115e268a2ec4445e9279b99a26c67c15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:36:51.912897 17954 ts_tablet_manager.cc:1905] T c47b5fc45d094a6c92f8573a8c59c5fd P 115e268a2ec4445e9279b99a26c67c15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:36:51.913761 17953 ts_tablet_manager.cc:1905] T c47b5fc45d094a6c92f8573a8c59c5fd P bb988f82fff54b6e8d468bd6075898b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:36:51.915670 17954 ts_tablet_manager.cc:1918] T c47b5fc45d094a6c92f8573a8c59c5fd P 115e268a2ec4445e9279b99a26c67c15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:36:51.915786 17954 log.cc:1196] T c47b5fc45d094a6c92f8573a8c59c5fd P 115e268a2ec4445e9279b99a26c67c15: Deleting WAL directory at /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/minicluster-data/ts-1/wal/wals/c47b5fc45d094a6c92f8573a8c59c5fd
I20260925 16:36:51.916096 17954 ts_tablet_manager.cc:1939] T c47b5fc45d094a6c92f8573a8c59c5fd P 115e268a2ec4445e9279b99a26c67c15: Deleting consensus metadata
I20260925 16:36:51.917438 17438 catalog_manager.cc:4841] TS 115e268a2ec4445e9279b99a26c67c15 (127.0.98.2:33139): tablet c47b5fc45d094a6c92f8573a8c59c5fd (table test-table [id=33d9177dfa5d4faf84482801c104b13a]) successfully deleted
I20260925 16:36:51.917750 17953 ts_tablet_manager.cc:1918] T c47b5fc45d094a6c92f8573a8c59c5fd P bb988f82fff54b6e8d468bd6075898b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:36:51.917901 17953 log.cc:1196] T c47b5fc45d094a6c92f8573a8c59c5fd P bb988f82fff54b6e8d468bd6075898b2: Deleting WAL directory at /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353966814071-392-0/minicluster-data/ts-2/wal/wals/c47b5fc45d094a6c92f8573a8c59c5fd
I20260925 16:36:51.918282 17953 ts_tablet_manager.cc:1939] T c47b5fc45d094a6c92f8573a8c59c5fd P bb988f82fff54b6e8d468bd6075898b2: Deleting consensus metadata
I20260925 16:36:51.919494 17439 catalog_manager.cc:4841] TS bb988f82fff54b6e8d468bd6075898b2 (127.0.98.3:44505): tablet c47b5fc45d094a6c92f8573a8c59c5fd (table test-table [id=33d9177dfa5d4faf84482801c104b13a]) successfully deleted
W20260925 16:36:51.920107 17438 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39077 (error 108)
I20260925 16:36:51.920360 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 17650
I20260925 16:36:51.921231 17926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:51.920850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39077 (local address 127.0.0.1:60258)
0925 16:36:51.921060 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:51.921133 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
W20260925 16:36:51.921396 17438 catalog_manager.cc:4584] TS 32654602b42b400cbc853b4422483634 (127.0.98.1:39077): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c47b5fc45d094a6c92f8573a8c59c5fd: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39077: connect: Connection refused (error 111)
I20260925 16:36:51.927352 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 17784
I20260925 16:36:51.932771 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 17422
2026-09-25T16:36:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2439 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-25T16:36:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:36:51Z Disabled control of system clock
I20260925 16:36:51.962812 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353966814071-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353966814071-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353966814071-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46575
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:46727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46575
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 16:36:52.070704 17959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:36:52.070979 17959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:36:52.071013 17959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:36:52.074892 17959 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:36:52.074947 17959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:36:52.074962 17959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:36:52.074980 17959 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:36:52.074994 17959 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:36:52.079620 17959 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353966814071-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353966814071-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46575
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353966814071-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:36:52.080651 17959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:36:52.081993 17959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:36:52.090153 17967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:36:52.090214 17964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:36:52.090193 17965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:36:52.090375 17959 server_base.cc:1034] running on GCE node
I20260925 16:36:52.091053 17959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:36:52.092108 17959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:36:52.093262 17959 hybrid_clock.cc:648] HybridClock initialized: now 1790354212093234 us; error 41 us; skew 500 ppm
I20260925 16:36:52.095090 17959 webserver.cc:460] Webserver started at http://127.0.98.62:38939/ using document root <none> and password file <none>
I20260925 16:36:52.095651 17959 fs_manager.cc:359] Metadata directory not provided
I20260925 16:36:52.095711 17959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:36:52.095949 17959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:36:52.097939 17959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353966814071-392-0/minicluster-data/master-0/data/instance:
uuid: "a83a2a0473fb41c7ac07a75719084455"
format_stamp: "Formatted at 2026-09-25 16:36:52 on dist-test-slave-65pf"
server_key: "709963a5dfc2974d117ab250d3d7f74d"
server_key_iv: "f69ba101c32514365885a239d7d2e6a0"
server_key_version: "encryptionkey@0"
I20260925 16:36:52.098489 17959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353966814071-392-0/minicluster-data/master-0/wal/instance:
uuid: "a83a2a0473fb41c7ac07a75719084455"
format_stamp: "Formatted at 2026-09-25 16:36:52 on dist-test-slave-65pf"
server_key: "709963a5dfc2974d117ab250d3d7f74d"
server_key_iv: "f69ba101c32514365885a239d7d2e6a0"
server_key_version: "encryptionkey@0"
I20260925 16:36:52.101976 17959 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 16:36:52.104295 17973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:52.105486 17959 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260925 16:36:52.105625 17959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353966814071-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353966814071-392-0/minicluster-data/master-0/wal
uuid: "a83a2a0473fb41c7ac07a75719084455"
format_stamp: "Formatted at 2026-09-25 16:36:52 on dist-test-slave-65pf"
server_key: "709963a5dfc2974d117ab250d3d7f74d"
server_key_iv: "f69ba101c32514365885a239d7d2e6a0"
server_key_version: "encryptionkey@0"
I20260925 16:36:52.105731 17959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353966814071-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353966814071-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353966814071-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:36:52.124591 17959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:36:52.125311 17959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:36:52.125542 17959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:36:52.132617 17959 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46575
I20260925 16:36:52.132644 18025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46575 every 8 connection(s)
I20260925 16:36:52.133543 17959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:36:52.136623 18026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:36:52.138270 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 17959
I20260925 16:36:52.138407 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353966814071-392-0/minicluster-data/master-0/wal/instance
I20260925 16:36:52.138728 392 external_mini_cluster.cc:1428] Setting key 5ab3498ff5e8bd673b50987af9fddd67
I20260925 16:36:52.143165 18026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a83a2a0473fb41c7ac07a75719084455: Bootstrap starting.
I20260925 16:36:52.145355 18026 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a83a2a0473fb41c7ac07a75719084455: Neither blocks nor log segments found. Creating new log.
I20260925 16:36:52.146242 18026 log.cc:824] T 00000000000000000000000000000000 P a83a2a0473fb41c7ac07a75719084455: Log is configured to *not* fsync() on all Append() calls
I20260925 16:36:52.147949 18029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:52.140144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50282 (local address 127.0.98.62:46575)
0925 16:36:52.140771 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:52.140785 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:52.140818 (+ 33us) server_negotiation.cc:407] Connection header received
0925 16:36:52.141458 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:52.141474 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:52.141911 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:52.142299 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:36:52.142766 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:52.143785 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:52.144665 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:52.145136 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:52.145314 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:52.146166 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:36:52.146170 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:36:52.146182 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:36:52.146661 (+ 479us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:36:52.146718 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:36:52.146738 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:36:52.146843 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:52.147000 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 16:36:52.147250 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":127,"threads_started":1}
I20260925 16:36:52.148306 18026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a83a2a0473fb41c7ac07a75719084455: No bootstrap required, opened a new log
I20260925 16:36:52.152362 18026 raft_consensus.cc:348] T 00000000000000000000000000000000 P a83a2a0473fb41c7ac07a75719084455 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a83a2a0473fb41c7ac07a75719084455" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46575 } }
I20260925 16:36:52.152639 18026 raft_consensus.cc:374] T 00000000000000000000000000000000 P a83a2a0473fb41c7ac07a75719084455 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:36:52.152709 18026 raft_consensus.cc:729] T 00000000000000000000000000000000 P a83a2a0473fb41c7ac07a75719084455 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a83a2a0473fb41c7ac07a75719084455, State: Initialized, Role: FOLLOWER
I20260925 16:36:52.153198 18026 consensus_queue.cc:260] T 00000000000000000000000000000000 P a83a2a0473fb41c7ac07a75719084455 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a83a2a0473fb41c7ac07a75719084455" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46575 } }
I20260925 16:36:52.153386 18026 raft_consensus.cc:388] T 00000000000000000000000000000000 P a83a2a0473fb41c7ac07a75719084455 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:36:52.153458 18026 raft_consensus.cc:482] T 00000000000000000000000000000000 P a83a2a0473fb41c7ac07a75719084455 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:36:52.153568 18026 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a83a2a0473fb41c7ac07a75719084455 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:36:52.154475 18026 raft_consensus.cc:504] T 00000000000000000000000000000000 P a83a2a0473fb41c7ac07a75719084455 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a83a2a0473fb41c7ac07a75719084455" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46575 } }
I20260925 16:36:52.154848 18026 leader_election.cc:302] T 00000000000000000000000000000000 P a83a2a0473fb41c7ac07a75719084455 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a83a2a0473fb41c7ac07a75719084455; no voters:
I20260925 16:36:52.155167 18026 leader_election.cc:288] T 00000000000000000000000000000000 P a83a2a0473fb41c7ac07a75719084455 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:36:52.155294 18031 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a83a2a0473fb41c7ac07a75719084455 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:36:52.155539 18031 raft_consensus.cc:686] T 00000000000000000000000000000000 P a83a2a0473fb41c7ac07a75719084455 [term 1 LEADER]: Becoming Leader. State: Replica: a83a2a0473fb41c7ac07a75719084455, State: Running, Role: LEADER
I20260925 16:36:52.155889 18031 consensus_queue.cc:237] T 00000000000000000000000000000000 P a83a2a0473fb41c7ac07a75719084455 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a83a2a0473fb41c7ac07a75719084455" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46575 } }
I20260925 16:36:52.156486 18026 sys_catalog.cc:564] T 00000000000000000000000000000000 P a83a2a0473fb41c7ac07a75719084455 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:36:52.157531 18032 sys_catalog.cc:455] T 00000000000000000000000000000000 P a83a2a0473fb41c7ac07a75719084455 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a83a2a0473fb41c7ac07a75719084455" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a83a2a0473fb41c7ac07a75719084455" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46575 } } }
I20260925 16:36:52.157791 18032 sys_catalog.cc:458] T 00000000000000000000000000000000 P a83a2a0473fb41c7ac07a75719084455 [sys.catalog]: This master's current role is: LEADER
I20260925 16:36:52.157531 18033 sys_catalog.cc:455] T 00000000000000000000000000000000 P a83a2a0473fb41c7ac07a75719084455 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a83a2a0473fb41c7ac07a75719084455. Latest consensus state: current_term: 1 leader_uuid: "a83a2a0473fb41c7ac07a75719084455" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a83a2a0473fb41c7ac07a75719084455" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46575 } } }
I20260925 16:36:52.158075 18033 sys_catalog.cc:458] T 00000000000000000000000000000000 P a83a2a0473fb41c7ac07a75719084455 [sys.catalog]: This master's current role is: LEADER
I20260925 16:36:52.158277 18040 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:36:52.161451 18040 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:36:52.167385 18040 catalog_manager.cc:1347] Generated new cluster ID: f644f3322c15445c85643b4fa3379552
I20260925 16:36:52.167462 18040 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:36:52.177310 18040 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:36:52.178431 18040 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:36:52.182889 18040 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a83a2a0473fb41c7ac07a75719084455: Generated new TSK 0
I20260925 16:36:52.183539 18040 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:36:52.191190 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353966814071-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353966814071-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353966814071-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46575
--builtin_ntp_servers=127.0.98.20:46727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 16:36:52.300829 18050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:36:52.301203 18050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:36:52.301270 18050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:36:52.305251 18050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:36:52.305330 18050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:36:52.305423 18050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 16:36:52.310348 18050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353966814071-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353966814071-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353966814071-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:36:52.311427 18050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:36:52.312752 18050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 16:36:52.322150 18050 server_base.cc:1034] running on GCE node
W20260925 16:36:52.322094 18056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:36:52.322099 18055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:36:52.322098 18058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:36:52.322790 18050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:36:52.323464 18050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:36:52.324648 18050 hybrid_clock.cc:648] HybridClock initialized: now 1790354212324631 us; error 31 us; skew 500 ppm
I20260925 16:36:52.326416 18050 webserver.cc:460] Webserver started at http://127.0.98.1:43929/ using document root <none> and password file <none>
I20260925 16:36:52.327050 18050 fs_manager.cc:359] Metadata directory not provided
I20260925 16:36:52.327140 18050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:36:52.327353 18050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:36:52.329298 18050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353966814071-392-0/minicluster-data/ts-0/data/instance:
uuid: "0aada1f307ce4175807e7f96e121df57"
format_stamp: "Formatted at 2026-09-25 16:36:52 on dist-test-slave-65pf"
server_key: "84e983aa73bec436a07f2099d11c380b"
server_key_iv: "a0412a0e2ed08c562917ecfba38fe4ae"
server_key_version: "encryptionkey@0"
I20260925 16:36:52.329882 18050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353966814071-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0aada1f307ce4175807e7f96e121df57"
format_stamp: "Formatted at 2026-09-25 16:36:52 on dist-test-slave-65pf"
server_key: "84e983aa73bec436a07f2099d11c380b"
server_key_iv: "a0412a0e2ed08c562917ecfba38fe4ae"
server_key_version: "encryptionkey@0"
I20260925 16:36:52.333416 18050 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 16:36:52.335852 18064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:52.337059 18050 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 16:36:52.337183 18050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353966814071-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353966814071-392-0/minicluster-data/ts-0/wal
uuid: "0aada1f307ce4175807e7f96e121df57"
format_stamp: "Formatted at 2026-09-25 16:36:52 on dist-test-slave-65pf"
server_key: "84e983aa73bec436a07f2099d11c380b"
server_key_iv: "a0412a0e2ed08c562917ecfba38fe4ae"
server_key_version: "encryptionkey@0"
I20260925 16:36:52.337292 18050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353966814071-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353966814071-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353966814071-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:36:52.358706 18050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:36:52.359650 18050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:36:52.359864 18050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:36:52.360661 18050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:36:52.361729 18050 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:36:52.361778 18050 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:52.361845 18050 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:36:52.361874 18050 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:52.370294 18050 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46143
I20260925 16:36:52.370355 18177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46143 every 8 connection(s)
I20260925 16:36:52.371299 18050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
I20260925 16:36:52.377604 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 18050
I20260925 16:36:52.377756 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353966814071-392-0/minicluster-data/ts-0/wal/instance
I20260925 16:36:52.378034 392 external_mini_cluster.cc:1428] Setting key aec3a9805994ee1c8a550ab3fb361221
I20260925 16:36:52.379941 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353966814071-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353966814071-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353966814071-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46575
--builtin_ntp_servers=127.0.98.20:46727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 16:36:52.380560 18029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:52.373071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41311 (local address 127.0.98.62:46575)
0925 16:36:52.373208 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:52.373211 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:52.374176 (+ 965us) server_negotiation.cc:407] Connection header received
0925 16:36:52.375634 (+ 1458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:52.375637 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:52.375697 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:52.375801 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:36:52.376689 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:52.377244 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:52.378535 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:52.378824 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:52.378940 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:52.380039 (+ 1099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:36:52.380042 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:36:52.380045 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:36:52.380060 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:36:52.380081 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:36:52.380090 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:36:52.380155 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:52.380353 (+ 198us) server_negotiation.cc:299] Negotiation successful
0925 16:36:52.380420 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":50}
I20260925 16:36:52.381616 18180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:52.373423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46575 (local address 127.0.98.1:41311)
0925 16:36:52.374031 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:52.374070 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 16:36:52.375423 (+ 1353us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:36:52.375974 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:36:52.375981 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:36:52.376062 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:36:52.376549 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:52.376561 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:52.377390 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:52.377394 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:52.378415 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:52.378424 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:52.378975 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:52.378979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:52.379147 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:52.379777 (+ 630us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 16:36:52.379779 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:36:52.379817 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 16:36:52.379880 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:36:52.380194 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:36:52.380202 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:36:52.380208 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 16:36:52.380346 (+ 138us) client_negotiation.cc:240] Negotiation successful
0925 16:36:52.380623 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":96,"threads_started":1}
I20260925 16:36:52.383002 18178 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46575
I20260925 16:36:52.383240 18178 heartbeater.cc:464] Registering TS with master...
I20260925 16:36:52.383868 18178 heartbeater.cc:511] Master 127.0.98.62:46575 requested a full tablet report, sending...
I20260925 16:36:52.385449 17990 ts_manager.cc:194] Registered new tserver with Master: 0aada1f307ce4175807e7f96e121df57 (127.0.98.1:46143)
I20260925 16:36:52.386633 17990 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:41311
W20260925 16:36:52.492594 18181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:36:52.492900 18181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:36:52.492971 18181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:36:52.496968 18181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:36:52.497049 18181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:36:52.497135 18181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 16:36:52.501816 18181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353966814071-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353966814071-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353966814071-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:36:52.502936 18181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:36:52.504268 18181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 16:36:52.513792 18181 server_base.cc:1034] running on GCE node
W20260925 16:36:52.513793 18186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:36:52.513746 18187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:36:52.513748 18189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:36:52.514314 18181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:36:52.514972 18181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:36:52.516136 18181 hybrid_clock.cc:648] HybridClock initialized: now 1790354212516121 us; error 25 us; skew 500 ppm
I20260925 16:36:52.517912 18181 webserver.cc:460] Webserver started at http://127.0.98.2:35051/ using document root <none> and password file <none>
I20260925 16:36:52.518481 18181 fs_manager.cc:359] Metadata directory not provided
I20260925 16:36:52.518540 18181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:36:52.518775 18181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:36:52.520766 18181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353966814071-392-0/minicluster-data/ts-1/data/instance:
uuid: "560573ac072b40999c519c5587c1f581"
format_stamp: "Formatted at 2026-09-25 16:36:52 on dist-test-slave-65pf"
server_key: "af62d5983fce526b5c5b7b839223af11"
server_key_iv: "9881652f795961808abf914e7174786a"
server_key_version: "encryptionkey@0"
I20260925 16:36:52.521304 18181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353966814071-392-0/minicluster-data/ts-1/wal/instance:
uuid: "560573ac072b40999c519c5587c1f581"
format_stamp: "Formatted at 2026-09-25 16:36:52 on dist-test-slave-65pf"
server_key: "af62d5983fce526b5c5b7b839223af11"
server_key_iv: "9881652f795961808abf914e7174786a"
server_key_version: "encryptionkey@0"
I20260925 16:36:52.524971 18181 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 16:36:52.527486 18195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:52.528759 18181 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:36:52.528906 18181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353966814071-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353966814071-392-0/minicluster-data/ts-1/wal
uuid: "560573ac072b40999c519c5587c1f581"
format_stamp: "Formatted at 2026-09-25 16:36:52 on dist-test-slave-65pf"
server_key: "af62d5983fce526b5c5b7b839223af11"
server_key_iv: "9881652f795961808abf914e7174786a"
server_key_version: "encryptionkey@0"
I20260925 16:36:52.529028 18181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353966814071-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353966814071-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353966814071-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:36:52.539028 18181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:36:52.539644 18181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:36:52.539842 18181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:36:52.540400 18181 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:36:52.541364 18181 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:36:52.541409 18181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:52.541471 18181 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:36:52.541507 18181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:52.549739 18181 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34489
I20260925 16:36:52.549767 18308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34489 every 8 connection(s)
I20260925 16:36:52.550565 18181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
I20260925 16:36:52.556531 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 18181
I20260925 16:36:52.556655 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353966814071-392-0/minicluster-data/ts-1/wal/instance
I20260925 16:36:52.556941 392 external_mini_cluster.cc:1428] Setting key 8548ffb215e47841767151a9b809853b
I20260925 16:36:52.559175 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353966814071-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353966814071-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353966814071-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46575
--builtin_ntp_servers=127.0.98.20:46727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 16:36:52.559938 18029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:52.552280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34341 (local address 127.0.98.62:46575)
0925 16:36:52.552418 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:52.552422 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:52.553361 (+ 939us) server_negotiation.cc:407] Connection header received
0925 16:36:52.554759 (+ 1398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:52.554762 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:52.554821 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:52.554926 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:36:52.555760 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:52.556370 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:52.557790 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:52.558024 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:52.558132 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:52.559427 (+ 1295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:36:52.559434 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:36:52.559439 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:36:52.559456 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:36:52.559480 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:36:52.559493 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:36:52.559555 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:52.559776 (+ 221us) server_negotiation.cc:299] Negotiation successful
0925 16:36:52.559833 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 16:36:52.561026 18311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:52.552631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46575 (local address 127.0.98.2:34341)
0925 16:36:52.553217 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:52.553254 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 16:36:52.554513 (+ 1259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:36:52.555087 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:36:52.555093 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:36:52.555173 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:36:52.555629 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:52.555640 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:52.556519 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:52.556522 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:52.557677 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:52.557686 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:52.558165 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:52.558169 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:52.558345 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:52.559120 (+ 775us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 16:36:52.559123 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:36:52.559173 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 16:36:52.559253 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:36:52.559620 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:36:52.559631 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:36:52.559639 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 16:36:52.559757 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 16:36:52.560070 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":98,"threads_started":1}
I20260925 16:36:52.562181 18309 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46575
I20260925 16:36:52.562399 18309 heartbeater.cc:464] Registering TS with master...
I20260925 16:36:52.562932 18309 heartbeater.cc:511] Master 127.0.98.62:46575 requested a full tablet report, sending...
I20260925 16:36:52.564029 17990 ts_manager.cc:194] Registered new tserver with Master: 560573ac072b40999c519c5587c1f581 (127.0.98.2:34489)
I20260925 16:36:52.564921 17990 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:34341
W20260925 16:36:52.672606 18312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:36:52.672838 18312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:36:52.672875 18312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:36:52.676826 18312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:36:52.676872 18312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:36:52.676921 18312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 16:36:52.681506 18312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353966814071-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353966814071-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353966814071-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:36:52.682510 18312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:36:52.683934 18312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:36:52.693163 18320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:36:52.693169 18317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:36:52.693264 18312 server_base.cc:1034] running on GCE node
W20260925 16:36:52.693202 18318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:36:52.693904 18312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:36:52.694490 18312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:36:52.695639 18312 hybrid_clock.cc:648] HybridClock initialized: now 1790354212695559 us; error 92 us; skew 500 ppm
I20260925 16:36:52.697361 18312 webserver.cc:460] Webserver started at http://127.0.98.3:44965/ using document root <none> and password file <none>
I20260925 16:36:52.697911 18312 fs_manager.cc:359] Metadata directory not provided
I20260925 16:36:52.697969 18312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:36:52.698207 18312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:36:52.700188 18312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353966814071-392-0/minicluster-data/ts-2/data/instance:
uuid: "7d574976e41e422897e69826765ee560"
format_stamp: "Formatted at 2026-09-25 16:36:52 on dist-test-slave-65pf"
server_key: "8e6a50945000660ede35c717bddef955"
server_key_iv: "7a42ba92174540f00f6d7bd24280ff3d"
server_key_version: "encryptionkey@0"
I20260925 16:36:52.700713 18312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353966814071-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7d574976e41e422897e69826765ee560"
format_stamp: "Formatted at 2026-09-25 16:36:52 on dist-test-slave-65pf"
server_key: "8e6a50945000660ede35c717bddef955"
server_key_iv: "7a42ba92174540f00f6d7bd24280ff3d"
server_key_version: "encryptionkey@0"
I20260925 16:36:52.704185 18312 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260925 16:36:52.706434 18326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:52.707695 18312 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:36:52.707825 18312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353966814071-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353966814071-392-0/minicluster-data/ts-2/wal
uuid: "7d574976e41e422897e69826765ee560"
format_stamp: "Formatted at 2026-09-25 16:36:52 on dist-test-slave-65pf"
server_key: "8e6a50945000660ede35c717bddef955"
server_key_iv: "7a42ba92174540f00f6d7bd24280ff3d"
server_key_version: "encryptionkey@0"
I20260925 16:36:52.707940 18312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353966814071-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353966814071-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353966814071-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:36:52.729815 18312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:36:52.730568 18312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:36:52.730830 18312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:36:52.731434 18312 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:36:52.732419 18312 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:36:52.732487 18312 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:52.732558 18312 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:36:52.732609 18312 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:52.741513 18312 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44999
I20260925 16:36:52.741559 18439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44999 every 8 connection(s)
I20260925 16:36:52.742394 18312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
I20260925 16:36:52.745857 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 18312
I20260925 16:36:52.745946 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353966814071-392-0/minicluster-data/ts-2/wal/instance
I20260925 16:36:52.746197 392 external_mini_cluster.cc:1428] Setting key a4407abe7a2a4c24f41fed3d97f4d37f
I20260925 16:36:52.751539 18029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:52.744181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41391 (local address 127.0.98.62:46575)
0925 16:36:52.744323 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:52.744326 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:52.745355 (+ 1029us) server_negotiation.cc:407] Connection header received
0925 16:36:52.746984 (+ 1629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:52.746987 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:52.747039 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:52.747113 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:36:52.747967 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:52.748516 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:52.749642 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:52.749874 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:52.749984 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:52.751055 (+ 1071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:36:52.751058 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:36:52.751061 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:36:52.751073 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:36:52.751101 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:36:52.751109 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:36:52.751185 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:52.751347 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 16:36:52.751404 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 16:36:52.752336 18442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:52.744544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46575 (local address 127.0.98.3:41391)
0925 16:36:52.745191 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:52.745238 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 16:36:52.746752 (+ 1514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:36:52.747278 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:36:52.747285 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:36:52.747364 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:36:52.747817 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:52.747829 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:52.748644 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:52.748647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:52.749516 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:52.749525 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:52.750020 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:52.750024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:52.750195 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:52.750811 (+ 616us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 16:36:52.750812 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:36:52.750849 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 16:36:52.750907 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:36:52.751197 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:36:52.751207 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:36:52.751220 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 16:36:52.751347 (+ 127us) client_negotiation.cc:240] Negotiation successful
0925 16:36:52.751604 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":119,"threads_started":1}
I20260925 16:36:52.753304 18440 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46575
I20260925 16:36:52.753530 18440 heartbeater.cc:464] Registering TS with master...
I20260925 16:36:52.753995 18440 heartbeater.cc:511] Master 127.0.98.62:46575 requested a full tablet report, sending...
I20260925 16:36:52.754961 17990 ts_manager.cc:194] Registered new tserver with Master: 7d574976e41e422897e69826765ee560 (127.0.98.3:44999)
I20260925 16:36:52.755466 17990 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:41391
I20260925 16:36:52.759835 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 16:36:52.765133 18029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:52.761613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50296 (local address 127.0.98.62:46575)
0925 16:36:52.761726 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:52.761729 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:52.761898 (+ 169us) server_negotiation.cc:407] Connection header received
0925 16:36:52.761979 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:52.761981 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:52.762015 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:52.762116 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:36:52.762420 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:52.762965 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:52.763792 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:52.763988 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:52.764056 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:52.764788 (+ 732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:36:52.764790 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:36:52.764792 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:36:52.764802 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:36:52.764817 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:36:52.764824 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:36:52.764906 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:52.764996 (+ 90us) server_negotiation.cc:299] Negotiation successful
0925 16:36:52.765039 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 16:36:52.767773 17990 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50296:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 16:36:52.770803 17990 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:36:52.784353 18451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:52.778971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44999 (local address 127.0.0.1:50826)
0925 16:36:52.779796 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:52.779807 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 16:36:52.779870 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:36:52.780585 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:36:52.780591 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:36:52.780612 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:36:52.780734 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:52.780742 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:52.781789 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:52.781793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:52.783020 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:52.783029 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:52.783841 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:52.783845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:52.783946 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:52.783985 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 16:36:52.784098 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 16:36:52.784204 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":775,"mutex_wait_us":1,"thread_start_us":102,"threads_started":1}
I20260925 16:36:52.784384 18454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:52.779354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50826 (local address 127.0.98.3:44999)
0925 16:36:52.780108 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:52.780119 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:52.780142 (+ 23us) server_negotiation.cc:407] Connection header received
0925 16:36:52.780214 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:52.780219 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:52.780433 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:52.780590 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:36:52.780875 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:52.781674 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:52.783137 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:52.783715 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:52.783827 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:52.784009 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:52.784126 (+ 117us) server_negotiation.cc:299] Negotiation successful
0925 16:36:52.784211 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":147,"threads_started":1}
I20260925 16:36:52.785266 18453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:52.778855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46143 (local address 127.0.0.1:46094)
0925 16:36:52.779647 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:52.779692 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 16:36:52.779805 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:36:52.780747 (+ 942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:36:52.780750 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:36:52.780765 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:36:52.780887 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:52.780894 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:52.782352 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:52.782358 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:52.783571 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:52.783581 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:52.784888 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:52.784892 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:52.784983 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:52.785004 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 16:36:52.785093 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 16:36:52.785154 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":79,"threads_started":1}
I20260925 16:36:52.785563 18455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:52.779535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34489 (local address 127.0.0.1:41548)
0925 16:36:52.780269 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:52.780289 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 16:36:52.780404 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:36:52.781037 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:36:52.781040 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:36:52.781056 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:36:52.781137 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:52.781143 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:52.782489 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:52.782492 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:52.783632 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:52.783642 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:52.784959 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:52.784962 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:52.785028 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:52.785045 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 16:36:52.785151 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 16:36:52.785219 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"mutex_wait_us":4,"thread_start_us":88,"threads_started":1}
I20260925 16:36:52.785619 18456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:52.779869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41548 (local address 127.0.98.2:34489)
0925 16:36:52.780177 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:52.780190 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:52.780382 (+ 192us) server_negotiation.cc:407] Connection header received
0925 16:36:52.780537 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:52.780544 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:52.780733 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:52.780876 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:36:52.781259 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:52.782383 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:52.783800 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:52.784610 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:52.784708 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:52.784894 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:52.785343 (+ 449us) server_negotiation.cc:299] Negotiation successful
0925 16:36:52.785453 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":131,"threads_started":1}
I20260925 16:36:52.785575 18452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:52.778978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46094 (local address 127.0.98.1:46143)
0925 16:36:52.780108 (+ 1130us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:52.780119 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:52.780142 (+ 23us) server_negotiation.cc:407] Connection header received
0925 16:36:52.780214 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:52.780219 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:52.780433 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:52.780590 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:36:52.781030 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:52.782223 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:52.783741 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:52.784535 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:52.784646 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:52.784814 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:52.785144 (+ 330us) server_negotiation.cc:299] Negotiation successful
0925 16:36:52.785231 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1030,"thread_start_us":90,"threads_started":1}
I20260925 16:36:52.787755 18374 tablet_service.cc:1467] Processing CreateTablet for tablet 8862cd12337843d79c28400e1dc5540a (DEFAULT_TABLE table=test-table [id=146e19221e97425dbe9a43738419334d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:36:52.788192 18112 tablet_service.cc:1467] Processing CreateTablet for tablet 8862cd12337843d79c28400e1dc5540a (DEFAULT_TABLE table=test-table [id=146e19221e97425dbe9a43738419334d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:36:52.788192 18243 tablet_service.cc:1467] Processing CreateTablet for tablet 8862cd12337843d79c28400e1dc5540a (DEFAULT_TABLE table=test-table [id=146e19221e97425dbe9a43738419334d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:36:52.788720 18374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8862cd12337843d79c28400e1dc5540a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:36:52.789189 18243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8862cd12337843d79c28400e1dc5540a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:36:52.789189 18112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8862cd12337843d79c28400e1dc5540a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:36:52.794584 18457 tablet_bootstrap.cc:492] T 8862cd12337843d79c28400e1dc5540a P 7d574976e41e422897e69826765ee560: Bootstrap starting.
I20260925 16:36:52.796553 18459 tablet_bootstrap.cc:492] T 8862cd12337843d79c28400e1dc5540a P 560573ac072b40999c519c5587c1f581: Bootstrap starting.
I20260925 16:36:52.797078 18457 tablet_bootstrap.cc:654] T 8862cd12337843d79c28400e1dc5540a P 7d574976e41e422897e69826765ee560: Neither blocks nor log segments found. Creating new log.
I20260925 16:36:52.797811 18457 log.cc:824] T 8862cd12337843d79c28400e1dc5540a P 7d574976e41e422897e69826765ee560: Log is configured to *not* fsync() on all Append() calls
I20260925 16:36:52.798287 18459 tablet_bootstrap.cc:654] T 8862cd12337843d79c28400e1dc5540a P 560573ac072b40999c519c5587c1f581: Neither blocks nor log segments found. Creating new log.
I20260925 16:36:52.798928 18458 tablet_bootstrap.cc:492] T 8862cd12337843d79c28400e1dc5540a P 0aada1f307ce4175807e7f96e121df57: Bootstrap starting.
I20260925 16:36:52.799144 18459 log.cc:824] T 8862cd12337843d79c28400e1dc5540a P 560573ac072b40999c519c5587c1f581: Log is configured to *not* fsync() on all Append() calls
I20260925 16:36:52.799716 18457 tablet_bootstrap.cc:492] T 8862cd12337843d79c28400e1dc5540a P 7d574976e41e422897e69826765ee560: No bootstrap required, opened a new log
I20260925 16:36:52.799926 18457 ts_tablet_manager.cc:1397] T 8862cd12337843d79c28400e1dc5540a P 7d574976e41e422897e69826765ee560: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260925 16:36:52.800983 18458 tablet_bootstrap.cc:654] T 8862cd12337843d79c28400e1dc5540a P 0aada1f307ce4175807e7f96e121df57: Neither blocks nor log segments found. Creating new log.
I20260925 16:36:52.801057 18459 tablet_bootstrap.cc:492] T 8862cd12337843d79c28400e1dc5540a P 560573ac072b40999c519c5587c1f581: No bootstrap required, opened a new log
I20260925 16:36:52.801231 18459 ts_tablet_manager.cc:1397] T 8862cd12337843d79c28400e1dc5540a P 560573ac072b40999c519c5587c1f581: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 16:36:52.801926 18458 log.cc:824] T 8862cd12337843d79c28400e1dc5540a P 0aada1f307ce4175807e7f96e121df57: Log is configured to *not* fsync() on all Append() calls
I20260925 16:36:52.803539 18457 raft_consensus.cc:348] T 8862cd12337843d79c28400e1dc5540a P 7d574976e41e422897e69826765ee560 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aada1f307ce4175807e7f96e121df57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46143 } } peers { permanent_uuid: "7d574976e41e422897e69826765ee560" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } } peers { permanent_uuid: "560573ac072b40999c519c5587c1f581" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34489 } }
I20260925 16:36:52.803761 18457 raft_consensus.cc:374] T 8862cd12337843d79c28400e1dc5540a P 7d574976e41e422897e69826765ee560 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:36:52.803804 18458 tablet_bootstrap.cc:492] T 8862cd12337843d79c28400e1dc5540a P 0aada1f307ce4175807e7f96e121df57: No bootstrap required, opened a new log
I20260925 16:36:52.803812 18457 raft_consensus.cc:729] T 8862cd12337843d79c28400e1dc5540a P 7d574976e41e422897e69826765ee560 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d574976e41e422897e69826765ee560, State: Initialized, Role: FOLLOWER
I20260925 16:36:52.803990 18458 ts_tablet_manager.cc:1397] T 8862cd12337843d79c28400e1dc5540a P 0aada1f307ce4175807e7f96e121df57: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 16:36:52.804273 18457 consensus_queue.cc:260] T 8862cd12337843d79c28400e1dc5540a P 7d574976e41e422897e69826765ee560 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aada1f307ce4175807e7f96e121df57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46143 } } peers { permanent_uuid: "7d574976e41e422897e69826765ee560" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } } peers { permanent_uuid: "560573ac072b40999c519c5587c1f581" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34489 } }
I20260925 16:36:52.804872 18440 heartbeater.cc:503] Master 127.0.98.62:46575 was elected leader, sending a full tablet report...
I20260925 16:36:52.805233 18457 ts_tablet_manager.cc:1428] T 8862cd12337843d79c28400e1dc5540a P 7d574976e41e422897e69826765ee560: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260925 16:36:52.805330 18459 raft_consensus.cc:348] T 8862cd12337843d79c28400e1dc5540a P 560573ac072b40999c519c5587c1f581 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aada1f307ce4175807e7f96e121df57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46143 } } peers { permanent_uuid: "7d574976e41e422897e69826765ee560" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } } peers { permanent_uuid: "560573ac072b40999c519c5587c1f581" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34489 } }
I20260925 16:36:52.805591 18459 raft_consensus.cc:374] T 8862cd12337843d79c28400e1dc5540a P 560573ac072b40999c519c5587c1f581 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:36:52.805645 18459 raft_consensus.cc:729] T 8862cd12337843d79c28400e1dc5540a P 560573ac072b40999c519c5587c1f581 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 560573ac072b40999c519c5587c1f581, State: Initialized, Role: FOLLOWER
I20260925 16:36:52.806084 18459 consensus_queue.cc:260] T 8862cd12337843d79c28400e1dc5540a P 560573ac072b40999c519c5587c1f581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aada1f307ce4175807e7f96e121df57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46143 } } peers { permanent_uuid: "7d574976e41e422897e69826765ee560" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } } peers { permanent_uuid: "560573ac072b40999c519c5587c1f581" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34489 } }
I20260925 16:36:52.806753 18309 heartbeater.cc:503] Master 127.0.98.62:46575 was elected leader, sending a full tablet report...
I20260925 16:36:52.807073 18459 ts_tablet_manager.cc:1428] T 8862cd12337843d79c28400e1dc5540a P 560573ac072b40999c519c5587c1f581: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 16:36:52.808138 18458 raft_consensus.cc:348] T 8862cd12337843d79c28400e1dc5540a P 0aada1f307ce4175807e7f96e121df57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aada1f307ce4175807e7f96e121df57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46143 } } peers { permanent_uuid: "7d574976e41e422897e69826765ee560" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } } peers { permanent_uuid: "560573ac072b40999c519c5587c1f581" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34489 } }
I20260925 16:36:52.808393 18458 raft_consensus.cc:374] T 8862cd12337843d79c28400e1dc5540a P 0aada1f307ce4175807e7f96e121df57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:36:52.808452 18458 raft_consensus.cc:729] T 8862cd12337843d79c28400e1dc5540a P 0aada1f307ce4175807e7f96e121df57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0aada1f307ce4175807e7f96e121df57, State: Initialized, Role: FOLLOWER
I20260925 16:36:52.808892 18458 consensus_queue.cc:260] T 8862cd12337843d79c28400e1dc5540a P 0aada1f307ce4175807e7f96e121df57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aada1f307ce4175807e7f96e121df57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46143 } } peers { permanent_uuid: "7d574976e41e422897e69826765ee560" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } } peers { permanent_uuid: "560573ac072b40999c519c5587c1f581" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34489 } }
I20260925 16:36:52.809556 18178 heartbeater.cc:503] Master 127.0.98.62:46575 was elected leader, sending a full tablet report...
I20260925 16:36:52.809854 18458 ts_tablet_manager.cc:1428] T 8862cd12337843d79c28400e1dc5540a P 0aada1f307ce4175807e7f96e121df57: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260925 16:36:52.872570 18179 tablet.cc:2369] T 8862cd12337843d79c28400e1dc5540a P 0aada1f307ce4175807e7f96e121df57: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:36:52.957055 18464 raft_consensus.cc:482] T 8862cd12337843d79c28400e1dc5540a P 560573ac072b40999c519c5587c1f581 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:36:52.957360 18464 raft_consensus.cc:504] T 8862cd12337843d79c28400e1dc5540a P 560573ac072b40999c519c5587c1f581 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aada1f307ce4175807e7f96e121df57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46143 } } peers { permanent_uuid: "7d574976e41e422897e69826765ee560" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } } peers { permanent_uuid: "560573ac072b40999c519c5587c1f581" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34489 } }
I20260925 16:36:52.958686 18464 leader_election.cc:288] T 8862cd12337843d79c28400e1dc5540a P 560573ac072b40999c519c5587c1f581 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0aada1f307ce4175807e7f96e121df57 (127.0.98.1:46143), 7d574976e41e422897e69826765ee560 (127.0.98.3:44999)
I20260925 16:36:52.962363 18311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:52.958885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46143 (local address 127.0.98.2:58657)
0925 16:36:52.959006 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:52.959024 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 16:36:52.959117 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:36:52.959433 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:36:52.959436 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:36:52.959462 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:36:52.959576 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:52.959581 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:52.960350 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:52.960353 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:52.961199 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:52.961215 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:52.961938 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:52.961942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:52.962041 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:52.962068 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 16:36:52.962154 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 16:36:52.962231 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":19,"spinlock_wait_cycles":6016}
I20260925 16:36:52.962361 18452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:52.959087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58657 (local address 127.0.98.1:46143)
0925 16:36:52.959214 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:52.959219 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:52.959241 (+ 22us) server_negotiation.cc:407] Connection header received
0925 16:36:52.959289 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:52.959292 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:52.959331 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:52.959423 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:36:52.959698 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:52.960204 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:52.961326 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:52.961819 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:52.961936 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:52.961995 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:52.962168 (+ 173us) server_negotiation.cc:299] Negotiation successful
0925 16:36:52.962231 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 16:36:52.962828 18466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:52.958957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44999 (local address 127.0.98.2:34217)
0925 16:36:52.959327 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:52.959345 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 16:36:52.959480 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:36:52.959764 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:36:52.959767 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:36:52.959781 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:36:52.959895 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:52.959900 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:52.960745 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:52.960751 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:52.961697 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:52.961706 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:52.962365 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:52.962369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:52.962476 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:52.962506 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 16:36:52.962615 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 16:36:52.962673 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":20,"spinlock_wait_cycles":3200,"thread_start_us":106,"threads_started":1}
I20260925 16:36:52.962850 18454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:52.959048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34217 (local address 127.0.98.3:44999)
0925 16:36:52.959169 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:52.959174 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:52.959430 (+ 256us) server_negotiation.cc:407] Connection header received
0925 16:36:52.959597 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:52.959600 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:52.959649 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:52.959739 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:36:52.959997 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:52.960638 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:52.961815 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:52.962230 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:52.962352 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:52.962405 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:52.962634 (+ 229us) server_negotiation.cc:299] Negotiation successful
0925 16:36:52.962717 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 16:36:52.963078 18132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8862cd12337843d79c28400e1dc5540a" candidate_uuid: "560573ac072b40999c519c5587c1f581" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0aada1f307ce4175807e7f96e121df57" is_pre_election: true
I20260925 16:36:52.963346 18132 raft_consensus.cc:2454] T 8862cd12337843d79c28400e1dc5540a P 0aada1f307ce4175807e7f96e121df57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 560573ac072b40999c519c5587c1f581 in term 0.
I20260925 16:36:52.963804 18198 leader_election.cc:302] T 8862cd12337843d79c28400e1dc5540a P 560573ac072b40999c519c5587c1f581 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0aada1f307ce4175807e7f96e121df57, 560573ac072b40999c519c5587c1f581; no voters:
I20260925 16:36:52.963943 18394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8862cd12337843d79c28400e1dc5540a" candidate_uuid: "560573ac072b40999c519c5587c1f581" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d574976e41e422897e69826765ee560" is_pre_election: true
I20260925 16:36:52.964088 18464 raft_consensus.cc:2781] T 8862cd12337843d79c28400e1dc5540a P 560573ac072b40999c519c5587c1f581 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:36:52.964200 18464 raft_consensus.cc:482] T 8862cd12337843d79c28400e1dc5540a P 560573ac072b40999c519c5587c1f581 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:36:52.964192 18394 raft_consensus.cc:2454] T 8862cd12337843d79c28400e1dc5540a P 7d574976e41e422897e69826765ee560 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 560573ac072b40999c519c5587c1f581 in term 0.
I20260925 16:36:52.964262 18464 raft_consensus.cc:3037] T 8862cd12337843d79c28400e1dc5540a P 560573ac072b40999c519c5587c1f581 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:36:52.965480 18464 raft_consensus.cc:504] T 8862cd12337843d79c28400e1dc5540a P 560573ac072b40999c519c5587c1f581 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aada1f307ce4175807e7f96e121df57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46143 } } peers { permanent_uuid: "7d574976e41e422897e69826765ee560" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } } peers { permanent_uuid: "560573ac072b40999c519c5587c1f581" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34489 } }
I20260925 16:36:52.965838 18464 leader_election.cc:288] T 8862cd12337843d79c28400e1dc5540a P 560573ac072b40999c519c5587c1f581 [CANDIDATE]: Term 1 election: Requested vote from peers 0aada1f307ce4175807e7f96e121df57 (127.0.98.1:46143), 7d574976e41e422897e69826765ee560 (127.0.98.3:44999)
I20260925 16:36:52.966172 18394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8862cd12337843d79c28400e1dc5540a" candidate_uuid: "560573ac072b40999c519c5587c1f581" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d574976e41e422897e69826765ee560"
I20260925 16:36:52.966293 18394 raft_consensus.cc:3037] T 8862cd12337843d79c28400e1dc5540a P 7d574976e41e422897e69826765ee560 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:36:52.966732 18132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8862cd12337843d79c28400e1dc5540a" candidate_uuid: "560573ac072b40999c519c5587c1f581" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0aada1f307ce4175807e7f96e121df57"
I20260925 16:36:52.966858 18132 raft_consensus.cc:3037] T 8862cd12337843d79c28400e1dc5540a P 0aada1f307ce4175807e7f96e121df57 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:36:52.967425 18394 raft_consensus.cc:2454] T 8862cd12337843d79c28400e1dc5540a P 7d574976e41e422897e69826765ee560 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 560573ac072b40999c519c5587c1f581 in term 1.
I20260925 16:36:52.967751 18199 leader_election.cc:302] T 8862cd12337843d79c28400e1dc5540a P 560573ac072b40999c519c5587c1f581 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 560573ac072b40999c519c5587c1f581, 7d574976e41e422897e69826765ee560; no voters:
I20260925 16:36:52.967926 18464 raft_consensus.cc:2781] T 8862cd12337843d79c28400e1dc5540a P 560573ac072b40999c519c5587c1f581 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:36:52.968062 18132 raft_consensus.cc:2454] T 8862cd12337843d79c28400e1dc5540a P 0aada1f307ce4175807e7f96e121df57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 560573ac072b40999c519c5587c1f581 in term 1.
I20260925 16:36:52.968166 18464 raft_consensus.cc:686] T 8862cd12337843d79c28400e1dc5540a P 560573ac072b40999c519c5587c1f581 [term 1 LEADER]: Becoming Leader. State: Replica: 560573ac072b40999c519c5587c1f581, State: Running, Role: LEADER
I20260925 16:36:52.968475 18464 consensus_queue.cc:237] T 8862cd12337843d79c28400e1dc5540a P 560573ac072b40999c519c5587c1f581 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aada1f307ce4175807e7f96e121df57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46143 } } peers { permanent_uuid: "7d574976e41e422897e69826765ee560" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } } peers { permanent_uuid: "560573ac072b40999c519c5587c1f581" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34489 } }
I20260925 16:36:52.971580 17990 catalog_manager.cc:5495] T 8862cd12337843d79c28400e1dc5540a P 560573ac072b40999c519c5587c1f581 reported cstate change: term changed from 0 to 1, leader changed from <none> to 560573ac072b40999c519c5587c1f581 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "560573ac072b40999c519c5587c1f581" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aada1f307ce4175807e7f96e121df57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d574976e41e422897e69826765ee560" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "560573ac072b40999c519c5587c1f581" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34489 } health_report { overall_health: HEALTHY } } }
W20260925 16:36:52.993317 18441 tablet.cc:2369] T 8862cd12337843d79c28400e1dc5540a P 7d574976e41e422897e69826765ee560: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:36:53.019207 18456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:53.015696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41562 (local address 127.0.98.2:34489)
0925 16:36:53.015878 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:53.015884 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:53.015899 (+ 15us) server_negotiation.cc:407] Connection header received
0925 16:36:53.015981 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:53.015983 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:53.016035 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:53.016145 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:36:53.016408 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:53.016911 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:53.017747 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:53.017968 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:53.018074 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:53.018374 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:36:53.018843 (+ 469us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:36:53.018977 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:53.019018 (+ 41us) server_negotiation.cc:299] Negotiation successful
0925 16:36:53.019065 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 16:36:53.025215 18132 raft_consensus.cc:1261] T 8862cd12337843d79c28400e1dc5540a P 0aada1f307ce4175807e7f96e121df57 [term 1 FOLLOWER]: Refusing update from remote peer 560573ac072b40999c519c5587c1f581: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:36:53.025215 18394 raft_consensus.cc:1261] T 8862cd12337843d79c28400e1dc5540a P 7d574976e41e422897e69826765ee560 [term 1 FOLLOWER]: Refusing update from remote peer 560573ac072b40999c519c5587c1f581: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:36:53.026057 18464 consensus_queue.cc:1035] T 8862cd12337843d79c28400e1dc5540a P 560573ac072b40999c519c5587c1f581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0aada1f307ce4175807e7f96e121df57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:36:53.026297 18467 consensus_queue.cc:1035] T 8862cd12337843d79c28400e1dc5540a P 560573ac072b40999c519c5587c1f581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d574976e41e422897e69826765ee560" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:36:53.033191 18471 mvcc.cc:205] Tried to move back new op lower bound from 7333290856543244288 to 7333290856321413120. Current Snapshot: MvccSnapshot[applied={T|T < 7333290856543244288}]
I20260925 16:36:53.033527 18472 mvcc.cc:205] Tried to move back new op lower bound from 7333290856543244288 to 7333290856321413120. Current Snapshot: MvccSnapshot[applied={T|T < 7333290856543244288}]
I20260925 16:36:53.033869 18470 mvcc.cc:205] Tried to move back new op lower bound from 7333290856543244288 to 7333290856321413120. Current Snapshot: MvccSnapshot[applied={T|T < 7333290856543244288}]
I20260925 16:36:53.042259 17988 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:36:53.042403 17988 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:36:53.044860 17988 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a83a2a0473fb41c7ac07a75719084455: Sending DeleteTablet for 3 replicas of tablet 8862cd12337843d79c28400e1dc5540a
I20260925 16:36:53.045610 18374 tablet_service.cc:1514] Processing DeleteTablet for tablet 8862cd12337843d79c28400e1dc5540a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:36:53 UTC) from {username='slave'} at 127.0.0.1:50826
I20260925 16:36:53.045681 18243 tablet_service.cc:1514] Processing DeleteTablet for tablet 8862cd12337843d79c28400e1dc5540a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:36:53 UTC) from {username='slave'} at 127.0.0.1:41548
I20260925 16:36:53.045778 18112 tablet_service.cc:1514] Processing DeleteTablet for tablet 8862cd12337843d79c28400e1dc5540a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:36:53 UTC) from {username='slave'} at 127.0.0.1:46094
I20260925 16:36:53.046180 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 18050
I20260925 16:36:53.050838 18480 tablet_replica.cc:330] T 8862cd12337843d79c28400e1dc5540a P 7d574976e41e422897e69826765ee560: stopping tablet replica
I20260925 16:36:53.050838 18481 tablet_replica.cc:330] T 8862cd12337843d79c28400e1dc5540a P 560573ac072b40999c519c5587c1f581: stopping tablet replica
I20260925 16:36:53.051184 18481 raft_consensus.cc:2229] T 8862cd12337843d79c28400e1dc5540a P 560573ac072b40999c519c5587c1f581 [term 1 LEADER]: Raft consensus shutting down.
I20260925 16:36:53.051184 18480 raft_consensus.cc:2229] T 8862cd12337843d79c28400e1dc5540a P 7d574976e41e422897e69826765ee560 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:36:53.051586 18480 raft_consensus.cc:2258] T 8862cd12337843d79c28400e1dc5540a P 7d574976e41e422897e69826765ee560 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:36:53.051625 18481 raft_consensus.cc:2258] T 8862cd12337843d79c28400e1dc5540a P 560573ac072b40999c519c5587c1f581 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:36:53.053099 18480 ts_tablet_manager.cc:1905] T 8862cd12337843d79c28400e1dc5540a P 7d574976e41e422897e69826765ee560: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:36:53.053411 18481 ts_tablet_manager.cc:1905] T 8862cd12337843d79c28400e1dc5540a P 560573ac072b40999c519c5587c1f581: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 16:36:53.053699 17976 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46143 (error 108)
I20260925 16:36:53.054477 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 18181
I20260925 16:36:53.055967 18480 ts_tablet_manager.cc:1918] T 8862cd12337843d79c28400e1dc5540a P 7d574976e41e422897e69826765ee560: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:36:53.056077 18480 log.cc:1196] T 8862cd12337843d79c28400e1dc5540a P 7d574976e41e422897e69826765ee560: Deleting WAL directory at /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353966814071-392-0/minicluster-data/ts-2/wal/wals/8862cd12337843d79c28400e1dc5540a
I20260925 16:36:53.056389 18480 ts_tablet_manager.cc:1939] T 8862cd12337843d79c28400e1dc5540a P 7d574976e41e422897e69826765ee560: Deleting consensus metadata
I20260925 16:36:53.057412 18453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:53.055544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46143 (local address 127.0.0.1:46100)
0925 16:36:53.057213 (+ 1669us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:53.057297 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1557}
I20260925 16:36:53.057536 17977 catalog_manager.cc:4841] TS 7d574976e41e422897e69826765ee560 (127.0.98.3:44999): tablet 8862cd12337843d79c28400e1dc5540a (table test-table [id=146e19221e97425dbe9a43738419334d]) successfully deleted
W20260925 16:36:53.057593 17976 catalog_manager.cc:4584] TS 0aada1f307ce4175807e7f96e121df57 (127.0.98.1:46143): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8862cd12337843d79c28400e1dc5540a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46143: connect: Connection refused (error 111)
I20260925 16:36:53.060983 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 18312
I20260925 16:36:53.061422 18453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:53.061127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34489 (local address 127.0.0.1:41576)
0925 16:36:53.061292 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:53.061341 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:34489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260925 16:36:53.066627 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 17959
2026-09-25T16:36:53Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1132 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:36:53 dist-test-slave-65pf krb5kdc[18486](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:36:53 dist-test-slave-65pf krb5kdc[18486](info): set up 2 sockets
Sep 25 16:36:53 dist-test-slave-65pf krb5kdc[18486](info): commencing operation
krb5kdc: starting...
W20260925 16:36:54.114776 392 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/krb5kdc/test-user.keytab.
Sep 25 16:36:54 dist-test-slave-65pf krb5kdc[18486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354214, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:36:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:36:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 16:36:54.234241 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33189
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33189
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:36:54.345021 18499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:36:54.345295 18499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:36:54.345337 18499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:36:54.349426 18499 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:36:54.349494 18499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:36:54.349516 18499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:36:54.349536 18499 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:36:54.349555 18499 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:36:54.355561 18499 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33189
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:36:54.356974 18499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:36:54.358392 18499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:36:54.368711 18505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:36:54.368744 18504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:36:54.368803 18507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:36:54.369086 18499 server_base.cc:1034] running on GCE node
I20260925 16:36:54.369688 18499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:36:54.370965 18499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:36:54.372133 18499 hybrid_clock.cc:648] HybridClock initialized: now 1790354214372115 us; error 28 us; skew 500 ppm
Sep 25 16:36:54 dist-test-slave-65pf krb5kdc[18486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354214, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:36:54.374972 18499 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 16:36:54.375828 18499 webserver.cc:460] Webserver started at http://127.0.98.62:41055/ using document root <none> and password file <none>
I20260925 16:36:54.376425 18499 fs_manager.cc:359] Metadata directory not provided
I20260925 16:36:54.376514 18499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:36:54.376742 18499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:36:54.378800 18499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/minicluster-data/master-0/data/instance:
uuid: "c860c6cccc314cb798aff469c10e0ec1"
format_stamp: "Formatted at 2026-09-25 16:36:54 on dist-test-slave-65pf"
server_key: "51c0657bcc76f595c09f5c7538126529"
server_key_iv: "2561c2ebc430b0511cb684b4e58bf044"
server_key_version: "encryptionkey@0"
I20260925 16:36:54.379357 18499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/minicluster-data/master-0/wal/instance:
uuid: "c860c6cccc314cb798aff469c10e0ec1"
format_stamp: "Formatted at 2026-09-25 16:36:54 on dist-test-slave-65pf"
server_key: "51c0657bcc76f595c09f5c7538126529"
server_key_iv: "2561c2ebc430b0511cb684b4e58bf044"
server_key_version: "encryptionkey@0"
I20260925 16:36:54.383086 18499 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 16:36:54.385468 18514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:54.386667 18499 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:36:54.386795 18499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/minicluster-data/master-0/wal
uuid: "c860c6cccc314cb798aff469c10e0ec1"
format_stamp: "Formatted at 2026-09-25 16:36:54 on dist-test-slave-65pf"
server_key: "51c0657bcc76f595c09f5c7538126529"
server_key_iv: "2561c2ebc430b0511cb684b4e58bf044"
server_key_version: "encryptionkey@0"
I20260925 16:36:54.386905 18499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:36:54.407114 18499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:36:54.409948 18499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:36:54.410187 18499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:36:54.417198 18499 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33189
I20260925 16:36:54.417232 18566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33189 every 8 connection(s)
I20260925 16:36:54.418133 18499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:36:54.420774 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 18499
I20260925 16:36:54.420928 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/minicluster-data/master-0/wal/instance
I20260925 16:36:54.421265 18567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:36:54.421290 392 external_mini_cluster.cc:1428] Setting key 7bea4f51e65cdfbfeab5765f12384f03
I20260925 16:36:54.428424 18567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c860c6cccc314cb798aff469c10e0ec1: Bootstrap starting.
Sep 25 16:36:54 dist-test-slave-65pf krb5kdc[18486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354214, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:36:54.431102 18567 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c860c6cccc314cb798aff469c10e0ec1: Neither blocks nor log segments found. Creating new log.
I20260925 16:36:54.431805 18567 log.cc:824] T 00000000000000000000000000000000 P c860c6cccc314cb798aff469c10e0ec1: Log is configured to *not* fsync() on all Append() calls
I20260925 16:36:54.433665 18567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c860c6cccc314cb798aff469c10e0ec1: No bootstrap required, opened a new log
I20260925 16:36:54.435586 18570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:54.422724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42328 (local address 127.0.98.62:33189)
0925 16:36:54.423124 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:54.423140 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:54.423174 (+ 34us) server_negotiation.cc:407] Connection header received
0925 16:36:54.423974 (+ 800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:54.423991 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:54.424304 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:54.424588 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:36:54.425566 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:54.426526 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:54.427753 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:54.428190 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:54.428302 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:54.431068 (+ 2766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:36:54.431073 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:36:54.431085 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:36:54.431110 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:36:54.432847 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:36:54.433291 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:36:54.433298 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:36:54.433317 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:36:54.433430 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:36:54.433707 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:36:54.433712 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:36:54.433714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:36:54.434134 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:36:54.434365 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:54.434629 (+ 264us) server_negotiation.cc:299] Negotiation successful
0925 16:36:54.434905 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":138,"threads_started":1}
I20260925 16:36:54.437491 18567 raft_consensus.cc:348] T 00000000000000000000000000000000 P c860c6cccc314cb798aff469c10e0ec1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c860c6cccc314cb798aff469c10e0ec1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33189 } }
I20260925 16:36:54.437816 18567 raft_consensus.cc:374] T 00000000000000000000000000000000 P c860c6cccc314cb798aff469c10e0ec1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:36:54.437881 18567 raft_consensus.cc:729] T 00000000000000000000000000000000 P c860c6cccc314cb798aff469c10e0ec1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c860c6cccc314cb798aff469c10e0ec1, State: Initialized, Role: FOLLOWER
I20260925 16:36:54.438311 18567 consensus_queue.cc:260] T 00000000000000000000000000000000 P c860c6cccc314cb798aff469c10e0ec1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c860c6cccc314cb798aff469c10e0ec1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33189 } }
I20260925 16:36:54.438427 18567 raft_consensus.cc:388] T 00000000000000000000000000000000 P c860c6cccc314cb798aff469c10e0ec1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:36:54.438473 18567 raft_consensus.cc:482] T 00000000000000000000000000000000 P c860c6cccc314cb798aff469c10e0ec1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:36:54.438541 18567 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c860c6cccc314cb798aff469c10e0ec1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:36:54.439394 18567 raft_consensus.cc:504] T 00000000000000000000000000000000 P c860c6cccc314cb798aff469c10e0ec1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c860c6cccc314cb798aff469c10e0ec1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33189 } }
I20260925 16:36:54.439678 18567 leader_election.cc:302] T 00000000000000000000000000000000 P c860c6cccc314cb798aff469c10e0ec1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c860c6cccc314cb798aff469c10e0ec1; no voters:
I20260925 16:36:54.439905 18567 leader_election.cc:288] T 00000000000000000000000000000000 P c860c6cccc314cb798aff469c10e0ec1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:36:54.440025 18572 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c860c6cccc314cb798aff469c10e0ec1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:36:54.440246 18572 raft_consensus.cc:686] T 00000000000000000000000000000000 P c860c6cccc314cb798aff469c10e0ec1 [term 1 LEADER]: Becoming Leader. State: Replica: c860c6cccc314cb798aff469c10e0ec1, State: Running, Role: LEADER
I20260925 16:36:54.440519 18572 consensus_queue.cc:237] T 00000000000000000000000000000000 P c860c6cccc314cb798aff469c10e0ec1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c860c6cccc314cb798aff469c10e0ec1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33189 } }
I20260925 16:36:54.441057 18567 sys_catalog.cc:564] T 00000000000000000000000000000000 P c860c6cccc314cb798aff469c10e0ec1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:36:54.442209 18573 sys_catalog.cc:455] T 00000000000000000000000000000000 P c860c6cccc314cb798aff469c10e0ec1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c860c6cccc314cb798aff469c10e0ec1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c860c6cccc314cb798aff469c10e0ec1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33189 } } }
I20260925 16:36:54.442335 18573 sys_catalog.cc:458] T 00000000000000000000000000000000 P c860c6cccc314cb798aff469c10e0ec1 [sys.catalog]: This master's current role is: LEADER
I20260925 16:36:54.442189 18574 sys_catalog.cc:455] T 00000000000000000000000000000000 P c860c6cccc314cb798aff469c10e0ec1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c860c6cccc314cb798aff469c10e0ec1. Latest consensus state: current_term: 1 leader_uuid: "c860c6cccc314cb798aff469c10e0ec1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c860c6cccc314cb798aff469c10e0ec1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33189 } } }
I20260925 16:36:54.442601 18574 sys_catalog.cc:458] T 00000000000000000000000000000000 P c860c6cccc314cb798aff469c10e0ec1 [sys.catalog]: This master's current role is: LEADER
I20260925 16:36:54.442811 18581 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:36:54.445586 18581 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:36:54.451073 18581 catalog_manager.cc:1347] Generated new cluster ID: b26343d9b9be42de883ec23ac9939ce0
I20260925 16:36:54.451153 18581 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:36:54.482080 18581 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:36:54.483153 18581 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:36:54.490677 18581 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c860c6cccc314cb798aff469c10e0ec1: Generated new TSK 0
I20260925 16:36:54.491539 18581 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 16:36:54.527710 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33189
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:36:54.636348 18593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:36:54.636585 18593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:36:54.636626 18593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:36:54.640570 18593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:36:54.640623 18593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:36:54.640717 18593 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 16:36:54.646126 18593 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:36:54.647161 18593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:36:54.648217 18593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:36:54.657467 18598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:36:54.657423 18601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:36:54.657511 18599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:36:54.657701 18593 server_base.cc:1034] running on GCE node
I20260925 16:36:54.658149 18593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:36:54.658907 18593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:36:54.660125 18593 hybrid_clock.cc:648] HybridClock initialized: now 1790354214660092 us; error 48 us; skew 500 ppm
Sep 25 16:36:54 dist-test-slave-65pf krb5kdc[18486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354214, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:36:54.667431 18593 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 16:36:54.668239 18593 webserver.cc:460] Webserver started at http://127.0.98.1:37911/ using document root <none> and password file <none>
I20260925 16:36:54.668788 18593 fs_manager.cc:359] Metadata directory not provided
I20260925 16:36:54.668846 18593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:36:54.669073 18593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:36:54.671060 18593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/minicluster-data/ts-0/data/instance:
uuid: "63a91deb00da41c0940784fe4e53c0f3"
format_stamp: "Formatted at 2026-09-25 16:36:54 on dist-test-slave-65pf"
server_key: "5b326481e509dd6fb9eaf0a77b45ce87"
server_key_iv: "18b4045a9d5b1cbc850cc8aa3c81706c"
server_key_version: "encryptionkey@0"
I20260925 16:36:54.671603 18593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/minicluster-data/ts-0/wal/instance:
uuid: "63a91deb00da41c0940784fe4e53c0f3"
format_stamp: "Formatted at 2026-09-25 16:36:54 on dist-test-slave-65pf"
server_key: "5b326481e509dd6fb9eaf0a77b45ce87"
server_key_iv: "18b4045a9d5b1cbc850cc8aa3c81706c"
server_key_version: "encryptionkey@0"
I20260925 16:36:54.675163 18593 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 16:36:54.677479 18608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:54.678668 18593 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:36:54.678787 18593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/minicluster-data/ts-0/wal
uuid: "63a91deb00da41c0940784fe4e53c0f3"
format_stamp: "Formatted at 2026-09-25 16:36:54 on dist-test-slave-65pf"
server_key: "5b326481e509dd6fb9eaf0a77b45ce87"
server_key_iv: "18b4045a9d5b1cbc850cc8aa3c81706c"
server_key_version: "encryptionkey@0"
I20260925 16:36:54.678900 18593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:36:54.694731 18593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:36:54.697372 18593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:36:54.697573 18593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:36:54.698113 18593 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:36:54.698956 18593 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:36:54.699024 18593 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:54.699090 18593 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:36:54.699139 18593 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:54.707597 18593 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39935
I20260925 16:36:54.707633 18721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39935 every 8 connection(s)
I20260925 16:36:54.708432 18593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
I20260925 16:36:54.713936 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 18593
I20260925 16:36:54.714068 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/minicluster-data/ts-0/wal/instance
I20260925 16:36:54.714347 392 external_mini_cluster.cc:1428] Setting key 71184eabcf23f74593c0da8d516fe4ad
Sep 25 16:36:54 dist-test-slave-65pf krb5kdc[18486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354214, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:36:54.722175 18570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:54.710151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54197 (local address 127.0.98.62:33189)
0925 16:36:54.710307 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:54.710312 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:54.711162 (+ 850us) server_negotiation.cc:407] Connection header received
0925 16:36:54.712380 (+ 1218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:54.712383 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:54.712428 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:54.712502 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:36:54.713582 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:54.714385 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:54.715690 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:54.716000 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:54.716117 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:54.718629 (+ 2512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:36:54.718633 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:36:54.718636 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:36:54.718667 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:36:54.720113 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:36:54.720653 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:36:54.720661 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:36:54.720662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:36:54.720735 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:36:54.721076 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:36:54.721081 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:36:54.721083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:36:54.721265 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:36:54.721375 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:54.721886 (+ 511us) server_negotiation.cc:299] Negotiation successful
0925 16:36:54.722012 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 16:36:54.723085 18724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:54.710516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33189 (local address 127.0.98.1:54197)
0925 16:36:54.711020 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:54.711061 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 16:36:54.712187 (+ 1126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:36:54.712665 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:36:54.712671 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:36:54.713004 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:36:54.713460 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:54.713473 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:54.714500 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:54.714503 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:54.715568 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:54.715580 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:54.716142 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:54.716145 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:54.716281 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:54.716876 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:36:54.716899 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:36:54.718435 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:36:54.720242 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:36:54.720248 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:36:54.720263 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:36:54.720537 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:36:54.720827 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:36:54.720832 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:36:54.720834 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:36:54.720978 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:36:54.721375 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:36:54.721383 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:36:54.721647 (+ 264us) client_negotiation.cc:769] Sending connection context
0925 16:36:54.721838 (+ 191us) client_negotiation.cc:240] Negotiation successful
0925 16:36:54.722169 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":119,"threads_started":1}
I20260925 16:36:54.724382 18722 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33189
I20260925 16:36:54.724627 18722 heartbeater.cc:464] Registering TS with master...
I20260925 16:36:54.725162 18722 heartbeater.cc:511] Master 127.0.98.62:33189 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260925 16:36:54.726691 18531 ts_manager.cc:194] Registered new tserver with Master: 63a91deb00da41c0940784fe4e53c0f3 (127.0.98.1:39935)
I20260925 16:36:54.727810 18531 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54197
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 16:36:54.742419 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33189
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:36:54.853202 18727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:36:54.853437 18727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:36:54.853494 18727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:36:54.857489 18727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:36:54.857576 18727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:36:54.857668 18727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 16:36:54.862898 18727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18727
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:36:54.863911 18727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:36:54.864912 18727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:36:54.874199 18732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:36:54.874199 18733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:36:54.874210 18735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:36:54.874416 18727 server_base.cc:1034] running on GCE node
I20260925 16:36:54.874941 18727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:36:54.875604 18727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:36:54.876785 18727 hybrid_clock.cc:648] HybridClock initialized: now 1790354214876767 us; error 30 us; skew 500 ppm
Sep 25 16:36:54 dist-test-slave-65pf krb5kdc[18486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354214, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:36:54.879470 18727 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 16:36:54.880245 18727 webserver.cc:460] Webserver started at http://127.0.98.2:38683/ using document root <none> and password file <none>
I20260925 16:36:54.880746 18727 fs_manager.cc:359] Metadata directory not provided
I20260925 16:36:54.880815 18727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:36:54.880976 18727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:36:54.883036 18727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/minicluster-data/ts-1/data/instance:
uuid: "09b8b0a91ae946b58421951383b4d4eb"
format_stamp: "Formatted at 2026-09-25 16:36:54 on dist-test-slave-65pf"
server_key: "085be9b7065f93feaaba183ef03ce194"
server_key_iv: "8a2043bc5c0115fc52095b461c42ea65"
server_key_version: "encryptionkey@0"
I20260925 16:36:54.883528 18727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/minicluster-data/ts-1/wal/instance:
uuid: "09b8b0a91ae946b58421951383b4d4eb"
format_stamp: "Formatted at 2026-09-25 16:36:54 on dist-test-slave-65pf"
server_key: "085be9b7065f93feaaba183ef03ce194"
server_key_iv: "8a2043bc5c0115fc52095b461c42ea65"
server_key_version: "encryptionkey@0"
I20260925 16:36:54.887072 18727 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 16:36:54.889477 18742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:54.890744 18727 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:36:54.890852 18727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/minicluster-data/ts-1/wal
uuid: "09b8b0a91ae946b58421951383b4d4eb"
format_stamp: "Formatted at 2026-09-25 16:36:54 on dist-test-slave-65pf"
server_key: "085be9b7065f93feaaba183ef03ce194"
server_key_iv: "8a2043bc5c0115fc52095b461c42ea65"
server_key_version: "encryptionkey@0"
I20260925 16:36:54.890949 18727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:36:54.920820 18727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:36:54.923779 18727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:36:54.923981 18727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:36:54.924556 18727 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:36:54.925386 18727 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:36:54.925447 18727 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:54.925554 18727 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:36:54.925593 18727 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:54.934305 18727 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33815
I20260925 16:36:54.934346 18855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33815 every 8 connection(s)
I20260925 16:36:54.935385 18727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
I20260925 16:36:54.939067 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 18727
I20260925 16:36:54.939178 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/minicluster-data/ts-1/wal/instance
I20260925 16:36:54.939471 392 external_mini_cluster.cc:1428] Setting key 2271c39d2c75b9d480903214da16cbbe
Sep 25 16:36:54 dist-test-slave-65pf krb5kdc[18486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354214, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260925 16:36:54.949560 18570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:54.937058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58837 (local address 127.0.98.62:33189)
0925 16:36:54.937192 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:54.937196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:54.938006 (+ 810us) server_negotiation.cc:407] Connection header received
0925 16:36:54.939607 (+ 1601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:54.939610 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:54.939658 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:54.939731 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:36:54.940897 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:54.941503 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:54.943166 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:54.943386 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:54.943506 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:54.946006 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:36:54.946010 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:36:54.946012 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:36:54.946033 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:36:54.947463 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:36:54.947945 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:36:54.947950 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:36:54.947952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:36:54.948027 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:36:54.948433 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:36:54.948438 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:36:54.948440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:36:54.948656 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:36:54.948764 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:54.949250 (+ 486us) server_negotiation.cc:299] Negotiation successful
0925 16:36:54.949364 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 16:36:54.950320 18858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:54.937434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33189 (local address 127.0.98.2:58837)
0925 16:36:54.937865 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:54.937908 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 16:36:54.939348 (+ 1440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:36:54.939927 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:36:54.939933 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:36:54.940282 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:36:54.940741 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:54.940779 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:54.941622 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:54.941629 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:54.943051 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:54.943060 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:54.943541 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:54.943545 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:54.943754 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:54.944394 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:36:54.944410 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:36:54.945845 (+ 1435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:36:54.947573 (+ 1728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:36:54.947579 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:36:54.947594 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:36:54.947839 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:36:54.948147 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:36:54.948152 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:36:54.948154 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:36:54.948319 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:36:54.948772 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:36:54.948781 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:36:54.949025 (+ 244us) client_negotiation.cc:769] Sending connection context
0925 16:36:54.949211 (+ 186us) client_negotiation.cc:240] Negotiation successful
0925 16:36:54.949528 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":119,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260925 16:36:54.951550 18856 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33189
I20260925 16:36:54.951794 18856 heartbeater.cc:464] Registering TS with master...
I20260925 16:36:54.952250 18856 heartbeater.cc:511] Master 127.0.98.62:33189 requested a full tablet report, sending...
I20260925 16:36:54.953227 18531 ts_manager.cc:194] Registered new tserver with Master: 09b8b0a91ae946b58421951383b4d4eb (127.0.98.2:33815)
I20260925 16:36:54.953751 18531 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58837
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 16:36:54.965732 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33189
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:36:55.073237 18861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:36:55.073464 18861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:36:55.073505 18861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:36:55.077512 18861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:36:55.077562 18861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:36:55.077647 18861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 16:36:55.082932 18861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18861
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:36:55.083930 18861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:36:55.084930 18861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:36:55.094334 18866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:36:55.094372 18867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:36:55.094352 18869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:36:55.094516 18861 server_base.cc:1034] running on GCE node
I20260925 16:36:55.095108 18861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:36:55.095726 18861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:36:55.096935 18861 hybrid_clock.cc:648] HybridClock initialized: now 1790354215096905 us; error 41 us; skew 500 ppm
Sep 25 16:36:55 dist-test-slave-65pf krb5kdc[18486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354215, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:36:55.099655 18861 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 16:36:55.100452 18861 webserver.cc:460] Webserver started at http://127.0.98.3:37355/ using document root <none> and password file <none>
I20260925 16:36:55.100986 18861 fs_manager.cc:359] Metadata directory not provided
I20260925 16:36:55.101043 18861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:36:55.101293 18861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:36:55.103302 18861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/minicluster-data/ts-2/data/instance:
uuid: "ad48a7044c2d422a83b37ed98ea9ab93"
format_stamp: "Formatted at 2026-09-25 16:36:55 on dist-test-slave-65pf"
server_key: "d8b20852e329e7b4ddee2a5e42670d73"
server_key_iv: "1cb4028093591d41499461dab130a7c5"
server_key_version: "encryptionkey@0"
I20260925 16:36:55.103838 18861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ad48a7044c2d422a83b37ed98ea9ab93"
format_stamp: "Formatted at 2026-09-25 16:36:55 on dist-test-slave-65pf"
server_key: "d8b20852e329e7b4ddee2a5e42670d73"
server_key_iv: "1cb4028093591d41499461dab130a7c5"
server_key_version: "encryptionkey@0"
I20260925 16:36:55.107365 18861 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 16:36:55.109596 18876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:55.110759 18861 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260925 16:36:55.110900 18861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/minicluster-data/ts-2/wal
uuid: "ad48a7044c2d422a83b37ed98ea9ab93"
format_stamp: "Formatted at 2026-09-25 16:36:55 on dist-test-slave-65pf"
server_key: "d8b20852e329e7b4ddee2a5e42670d73"
server_key_iv: "1cb4028093591d41499461dab130a7c5"
server_key_version: "encryptionkey@0"
I20260925 16:36:55.110996 18861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:36:55.127465 18861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:36:55.130714 18861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:36:55.130934 18861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:36:55.131570 18861 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:36:55.132483 18861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:36:55.132560 18861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:55.132639 18861 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:36:55.132691 18861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:55.142109 18861 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40789
I20260925 16:36:55.142171 18989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40789 every 8 connection(s)
I20260925 16:36:55.143134 18861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
I20260925 16:36:55.151499 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 18861
I20260925 16:36:55.151711 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/minicluster-data/ts-2/wal/instance
I20260925 16:36:55.152040 392 external_mini_cluster.cc:1428] Setting key f2982278c903cd9ef7c40074684d2759
Sep 25 16:36:55 dist-test-slave-65pf krb5kdc[18486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354215, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:36:55.157009 18570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:55.144920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52819 (local address 127.0.98.62:33189)
0925 16:36:55.145057 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:55.145061 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:55.145908 (+ 847us) server_negotiation.cc:407] Connection header received
0925 16:36:55.147270 (+ 1362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:55.147274 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:55.147332 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:55.147434 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:36:55.148581 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:55.149224 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:55.150380 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:55.150702 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:55.150819 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:55.153351 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:36:55.153354 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:36:55.153357 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:36:55.153384 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:36:55.154986 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:36:55.155502 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:36:55.155507 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:36:55.155509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:36:55.155594 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:36:55.155930 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:36:55.155933 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:36:55.155935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:36:55.156115 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:36:55.156219 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:55.156710 (+ 491us) server_negotiation.cc:299] Negotiation successful
0925 16:36:55.156846 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 16:36:55.157852 18992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:55.145309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33189 (local address 127.0.98.3:52819)
0925 16:36:55.145768 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:55.145810 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 16:36:55.147048 (+ 1238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:36:55.147598 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:36:55.147604 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:36:55.147951 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:36:55.148415 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:55.148427 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:55.149363 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:55.149367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:55.150247 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:55.150255 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:55.150857 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:55.150861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:55.151036 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:55.151682 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:36:55.151706 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:36:55.153192 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:36:55.155121 (+ 1929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:36:55.155127 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:36:55.155141 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:36:55.155401 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:36:55.155690 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:36:55.155693 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:36:55.155695 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:36:55.155830 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:36:55.156215 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:36:55.156222 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:36:55.156474 (+ 252us) client_negotiation.cc:769] Sending connection context
0925 16:36:55.156657 (+ 183us) client_negotiation.cc:240] Negotiation successful
0925 16:36:55.156983 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":161,"threads_started":1}
I20260925 16:36:55.159036 18990 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33189
I20260925 16:36:55.159266 18990 heartbeater.cc:464] Registering TS with master...
I20260925 16:36:55.159737 18990 heartbeater.cc:511] Master 127.0.98.62:33189 requested a full tablet report, sending...
I20260925 16:36:55.160784 18531 ts_manager.cc:194] Registered new tserver with Master: ad48a7044c2d422a83b37ed98ea9ab93 (127.0.98.3:40789)
I20260925 16:36:55.161355 18531 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52819
I20260925 16:36:55.165917 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 16:36:55.175467 18570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:55.167917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42340 (local address 127.0.98.62:33189)
0925 16:36:55.168082 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:55.168087 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:55.168236 (+ 149us) server_negotiation.cc:407] Connection header received
0925 16:36:55.168378 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:55.168382 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:55.168425 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:55.168519 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:36:55.169232 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:55.169742 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:55.170592 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:55.170831 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:55.170941 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:55.172436 (+ 1495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:36:55.172440 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:36:55.172442 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:36:55.172466 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:36:55.173581 (+ 1115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:36:55.174088 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:36:55.174092 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:36:55.174094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:36:55.174174 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:36:55.174503 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:36:55.174506 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:36:55.174508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:36:55.174768 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:36:55.174929 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:55.175201 (+ 272us) server_negotiation.cc:299] Negotiation successful
0925 16:36:55.175304 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 16:36:55.178264 18531 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42340:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 16:36:55.181288 18531 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:36:55.195775 19005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:55.189970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45664 (local address 127.0.98.2:33815)
0925 16:36:55.190431 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:55.190445 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:55.190464 (+ 19us) server_negotiation.cc:407] Connection header received
0925 16:36:55.190541 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:55.190550 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:55.190790 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:55.191334 (+ 544us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:36:55.191490 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:55.192624 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:55.194148 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:55.194982 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:55.195108 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:55.195283 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:55.195486 (+ 203us) server_negotiation.cc:299] Negotiation successful
0925 16:36:55.195597 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":141,"threads_started":1}
I20260925 16:36:55.195741 19001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:55.189250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33815 (local address 127.0.0.1:45664)
0925 16:36:55.189711 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:55.189770 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 16:36:55.189909 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:36:55.191007 (+ 1098us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:36:55.191012 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:36:55.191035 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:36:55.191177 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:55.191185 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:55.192765 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:55.192769 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:55.194004 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:55.194011 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:55.195091 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:55.195094 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:55.195263 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:55.195319 (+ 56us) client_negotiation.cc:769] Sending connection context
0925 16:36:55.195452 (+ 133us) client_negotiation.cc:240] Negotiation successful
0925 16:36:55.195597 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":110,"thread_start_us":94,"threads_started":1}
I20260925 16:36:55.195763 19004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:55.189354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39935 (local address 127.0.0.1:41690)
0925 16:36:55.190356 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:55.190369 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:36:55.190520 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:36:55.191190 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:36:55.191192 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:36:55.191209 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:36:55.191294 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:55.191299 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:55.192807 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:55.192809 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:55.193965 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:55.193974 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:55.195356 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:55.195359 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:55.195461 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:55.195484 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 16:36:55.195555 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 16:36:55.195620 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":939,"mutex_wait_us":50,"thread_start_us":101,"threads_started":1}
I20260925 16:36:55.196087 19003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:55.189514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41690 (local address 127.0.98.1:39935)
0925 16:36:55.190216 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:55.190231 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:55.190449 (+ 218us) server_negotiation.cc:407] Connection header received
0925 16:36:55.190698 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:55.190702 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:55.190868 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:55.190998 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:36:55.191409 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:55.192624 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:55.194091 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:55.195053 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:55.195143 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:55.195283 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:55.195832 (+ 549us) server_negotiation.cc:299] Negotiation successful
0925 16:36:55.195937 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":99,"threads_started":1}
I20260925 16:36:55.196218 19006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:55.190065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60658 (local address 127.0.98.3:40789)
0925 16:36:55.190816 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:55.190830 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:55.190864 (+ 34us) server_negotiation.cc:407] Connection header received
0925 16:36:55.190936 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:55.190944 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:55.191115 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:55.191263 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:36:55.191473 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:55.192208 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:55.193943 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:55.194698 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:55.194807 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:55.194986 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:55.195969 (+ 983us) server_negotiation.cc:299] Negotiation successful
0925 16:36:55.196060 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":91,"threads_started":1}
I20260925 16:36:55.196751 19002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:55.189935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40789 (local address 127.0.0.1:60658)
0925 16:36:55.190461 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:55.190474 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:36:55.190558 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:36:55.191251 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:36:55.191254 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:36:55.191271 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:36:55.191381 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:55.191390 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:55.192352 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:55.192355 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:55.193788 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:55.193798 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:55.195071 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:55.195076 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:55.195255 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:55.195319 (+ 64us) client_negotiation.cc:769] Sending connection context
0925 16:36:55.195438 (+ 119us) client_negotiation.cc:240] Negotiation successful
0925 16:36:55.196603 (+ 1165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"mutex_wait_us":76,"thread_start_us":94,"threads_started":1}
I20260925 16:36:55.199245 18790 tablet_service.cc:1467] Processing CreateTablet for tablet ccb9ec462b514d528ebbad653913963c (DEFAULT_TABLE table=test-table [id=61c32c95684841fbaa6f86b4cc594e53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:36:55.199246 18656 tablet_service.cc:1467] Processing CreateTablet for tablet ccb9ec462b514d528ebbad653913963c (DEFAULT_TABLE table=test-table [id=61c32c95684841fbaa6f86b4cc594e53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:36:55.199414 18924 tablet_service.cc:1467] Processing CreateTablet for tablet ccb9ec462b514d528ebbad653913963c (DEFAULT_TABLE table=test-table [id=61c32c95684841fbaa6f86b4cc594e53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:36:55.200294 18924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccb9ec462b514d528ebbad653913963c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:36:55.200314 18790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccb9ec462b514d528ebbad653913963c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:36:55.200314 18656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccb9ec462b514d528ebbad653913963c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:36:55.206357 19007 tablet_bootstrap.cc:492] T ccb9ec462b514d528ebbad653913963c P ad48a7044c2d422a83b37ed98ea9ab93: Bootstrap starting.
I20260925 16:36:55.208019 19009 tablet_bootstrap.cc:492] T ccb9ec462b514d528ebbad653913963c P 09b8b0a91ae946b58421951383b4d4eb: Bootstrap starting.
I20260925 16:36:55.208019 19008 tablet_bootstrap.cc:492] T ccb9ec462b514d528ebbad653913963c P 63a91deb00da41c0940784fe4e53c0f3: Bootstrap starting.
I20260925 16:36:55.209031 19007 tablet_bootstrap.cc:654] T ccb9ec462b514d528ebbad653913963c P ad48a7044c2d422a83b37ed98ea9ab93: Neither blocks nor log segments found. Creating new log.
I20260925 16:36:55.210135 19008 tablet_bootstrap.cc:654] T ccb9ec462b514d528ebbad653913963c P 63a91deb00da41c0940784fe4e53c0f3: Neither blocks nor log segments found. Creating new log.
I20260925 16:36:55.210135 19009 tablet_bootstrap.cc:654] T ccb9ec462b514d528ebbad653913963c P 09b8b0a91ae946b58421951383b4d4eb: Neither blocks nor log segments found. Creating new log.
I20260925 16:36:55.210155 19007 log.cc:824] T ccb9ec462b514d528ebbad653913963c P ad48a7044c2d422a83b37ed98ea9ab93: Log is configured to *not* fsync() on all Append() calls
I20260925 16:36:55.211407 19008 log.cc:824] T ccb9ec462b514d528ebbad653913963c P 63a91deb00da41c0940784fe4e53c0f3: Log is configured to *not* fsync() on all Append() calls
I20260925 16:36:55.211407 19009 log.cc:824] T ccb9ec462b514d528ebbad653913963c P 09b8b0a91ae946b58421951383b4d4eb: Log is configured to *not* fsync() on all Append() calls
I20260925 16:36:55.212106 19007 tablet_bootstrap.cc:492] T ccb9ec462b514d528ebbad653913963c P ad48a7044c2d422a83b37ed98ea9ab93: No bootstrap required, opened a new log
I20260925 16:36:55.212322 19007 ts_tablet_manager.cc:1397] T ccb9ec462b514d528ebbad653913963c P ad48a7044c2d422a83b37ed98ea9ab93: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 16:36:55.213394 19009 tablet_bootstrap.cc:492] T ccb9ec462b514d528ebbad653913963c P 09b8b0a91ae946b58421951383b4d4eb: No bootstrap required, opened a new log
I20260925 16:36:55.213567 19009 ts_tablet_manager.cc:1397] T ccb9ec462b514d528ebbad653913963c P 09b8b0a91ae946b58421951383b4d4eb: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260925 16:36:55.213714 19008 tablet_bootstrap.cc:492] T ccb9ec462b514d528ebbad653913963c P 63a91deb00da41c0940784fe4e53c0f3: No bootstrap required, opened a new log
I20260925 16:36:55.213907 19008 ts_tablet_manager.cc:1397] T ccb9ec462b514d528ebbad653913963c P 63a91deb00da41c0940784fe4e53c0f3: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 16:36:55.215914 19007 raft_consensus.cc:348] T ccb9ec462b514d528ebbad653913963c P ad48a7044c2d422a83b37ed98ea9ab93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b8b0a91ae946b58421951383b4d4eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33815 } } peers { permanent_uuid: "63a91deb00da41c0940784fe4e53c0f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39935 } } peers { permanent_uuid: "ad48a7044c2d422a83b37ed98ea9ab93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40789 } }
I20260925 16:36:55.216176 19007 raft_consensus.cc:374] T ccb9ec462b514d528ebbad653913963c P ad48a7044c2d422a83b37ed98ea9ab93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:36:55.216243 19007 raft_consensus.cc:729] T ccb9ec462b514d528ebbad653913963c P ad48a7044c2d422a83b37ed98ea9ab93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad48a7044c2d422a83b37ed98ea9ab93, State: Initialized, Role: FOLLOWER
I20260925 16:36:55.216706 19007 consensus_queue.cc:260] T ccb9ec462b514d528ebbad653913963c P ad48a7044c2d422a83b37ed98ea9ab93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b8b0a91ae946b58421951383b4d4eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33815 } } peers { permanent_uuid: "63a91deb00da41c0940784fe4e53c0f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39935 } } peers { permanent_uuid: "ad48a7044c2d422a83b37ed98ea9ab93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40789 } }
I20260925 16:36:55.217347 18990 heartbeater.cc:503] Master 127.0.98.62:33189 was elected leader, sending a full tablet report...
I20260925 16:36:55.217666 19007 ts_tablet_manager.cc:1428] T ccb9ec462b514d528ebbad653913963c P ad48a7044c2d422a83b37ed98ea9ab93: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 16:36:55.217502 19009 raft_consensus.cc:348] T ccb9ec462b514d528ebbad653913963c P 09b8b0a91ae946b58421951383b4d4eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b8b0a91ae946b58421951383b4d4eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33815 } } peers { permanent_uuid: "63a91deb00da41c0940784fe4e53c0f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39935 } } peers { permanent_uuid: "ad48a7044c2d422a83b37ed98ea9ab93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40789 } }
I20260925 16:36:55.217772 19009 raft_consensus.cc:374] T ccb9ec462b514d528ebbad653913963c P 09b8b0a91ae946b58421951383b4d4eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:36:55.217844 19009 raft_consensus.cc:729] T ccb9ec462b514d528ebbad653913963c P 09b8b0a91ae946b58421951383b4d4eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09b8b0a91ae946b58421951383b4d4eb, State: Initialized, Role: FOLLOWER
I20260925 16:36:55.218078 19008 raft_consensus.cc:348] T ccb9ec462b514d528ebbad653913963c P 63a91deb00da41c0940784fe4e53c0f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b8b0a91ae946b58421951383b4d4eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33815 } } peers { permanent_uuid: "63a91deb00da41c0940784fe4e53c0f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39935 } } peers { permanent_uuid: "ad48a7044c2d422a83b37ed98ea9ab93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40789 } }
I20260925 16:36:55.218287 19008 raft_consensus.cc:374] T ccb9ec462b514d528ebbad653913963c P 63a91deb00da41c0940784fe4e53c0f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:36:55.218333 19008 raft_consensus.cc:729] T ccb9ec462b514d528ebbad653913963c P 63a91deb00da41c0940784fe4e53c0f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63a91deb00da41c0940784fe4e53c0f3, State: Initialized, Role: FOLLOWER
I20260925 16:36:55.218267 19009 consensus_queue.cc:260] T ccb9ec462b514d528ebbad653913963c P 09b8b0a91ae946b58421951383b4d4eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b8b0a91ae946b58421951383b4d4eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33815 } } peers { permanent_uuid: "63a91deb00da41c0940784fe4e53c0f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39935 } } peers { permanent_uuid: "ad48a7044c2d422a83b37ed98ea9ab93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40789 } }
I20260925 16:36:55.218918 18856 heartbeater.cc:503] Master 127.0.98.62:33189 was elected leader, sending a full tablet report...
I20260925 16:36:55.219201 19009 ts_tablet_manager.cc:1428] T ccb9ec462b514d528ebbad653913963c P 09b8b0a91ae946b58421951383b4d4eb: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260925 16:36:55.219798 19008 consensus_queue.cc:260] T ccb9ec462b514d528ebbad653913963c P 63a91deb00da41c0940784fe4e53c0f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b8b0a91ae946b58421951383b4d4eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33815 } } peers { permanent_uuid: "63a91deb00da41c0940784fe4e53c0f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39935 } } peers { permanent_uuid: "ad48a7044c2d422a83b37ed98ea9ab93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40789 } }
I20260925 16:36:55.221297 18722 heartbeater.cc:503] Master 127.0.98.62:33189 was elected leader, sending a full tablet report...
I20260925 16:36:55.221451 19008 ts_tablet_manager.cc:1428] T ccb9ec462b514d528ebbad653913963c P 63a91deb00da41c0940784fe4e53c0f3: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260925 16:36:55.319324 19015 raft_consensus.cc:482] T ccb9ec462b514d528ebbad653913963c P 63a91deb00da41c0940784fe4e53c0f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:36:55.319540 19015 raft_consensus.cc:504] T ccb9ec462b514d528ebbad653913963c P 63a91deb00da41c0940784fe4e53c0f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b8b0a91ae946b58421951383b4d4eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33815 } } peers { permanent_uuid: "63a91deb00da41c0940784fe4e53c0f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39935 } } peers { permanent_uuid: "ad48a7044c2d422a83b37ed98ea9ab93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40789 } }
I20260925 16:36:55.320590 19015 leader_election.cc:288] T ccb9ec462b514d528ebbad653913963c P 63a91deb00da41c0940784fe4e53c0f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09b8b0a91ae946b58421951383b4d4eb (127.0.98.2:33815), ad48a7044c2d422a83b37ed98ea9ab93 (127.0.98.3:40789)
I20260925 16:36:55.324483 19016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:55.320879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40789 (local address 127.0.98.1:37793)
0925 16:36:55.321242 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:55.321255 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:36:55.321352 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:36:55.321640 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:36:55.321645 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:36:55.321664 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:36:55.321802 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:55.321807 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:55.322531 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:55.322536 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:55.323373 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:55.323382 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:55.324037 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:55.324044 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:55.324138 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:55.324172 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 16:36:55.324246 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 16:36:55.324300 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":1,"thread_start_us":91,"threads_started":1}
I20260925 16:36:55.324491 19006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:55.320991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37793 (local address 127.0.98.3:40789)
0925 16:36:55.321119 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:55.321124 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:55.321348 (+ 224us) server_negotiation.cc:407] Connection header received
0925 16:36:55.321466 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:55.321470 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:55.321529 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:55.321601 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:36:55.321908 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:55.322417 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:55.323502 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:55.323911 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:55.323990 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:55.324068 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:55.324287 (+ 219us) server_negotiation.cc:299] Negotiation successful
0925 16:36:55.324371 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 16:36:55.324566 19005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:55.320982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35649 (local address 127.0.98.2:33815)
0925 16:36:55.321113 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:55.321118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:55.321458 (+ 340us) server_negotiation.cc:407] Connection header received
0925 16:36:55.321527 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:55.321529 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:55.321569 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:55.321664 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:36:55.321958 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:55.322440 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:55.323555 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:55.323973 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:55.324078 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:55.324140 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:55.324371 (+ 231us) server_negotiation.cc:299] Negotiation successful
0925 16:36:55.324441 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 16:36:55.324532 19017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:55.320879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33815 (local address 127.0.98.1:35649)
0925 16:36:55.321294 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:55.321330 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 16:36:55.321450 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:36:55.321698 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:36:55.321702 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:36:55.321718 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:36:55.321813 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:55.321817 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:55.322618 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:55.322625 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:55.323399 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:55.323407 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:55.324103 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:55.324107 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:55.324213 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:55.324256 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 16:36:55.324333 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 16:36:55.324395 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"mutex_wait_us":36,"thread_start_us":111,"threads_started":1}
I20260925 16:36:55.325145 18944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccb9ec462b514d528ebbad653913963c" candidate_uuid: "63a91deb00da41c0940784fe4e53c0f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad48a7044c2d422a83b37ed98ea9ab93" is_pre_election: true
I20260925 16:36:55.325350 18810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccb9ec462b514d528ebbad653913963c" candidate_uuid: "63a91deb00da41c0940784fe4e53c0f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09b8b0a91ae946b58421951383b4d4eb" is_pre_election: true
I20260925 16:36:55.325465 18944 raft_consensus.cc:2454] T ccb9ec462b514d528ebbad653913963c P ad48a7044c2d422a83b37ed98ea9ab93 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63a91deb00da41c0940784fe4e53c0f3 in term 0.
I20260925 16:36:55.325654 18810 raft_consensus.cc:2454] T ccb9ec462b514d528ebbad653913963c P 09b8b0a91ae946b58421951383b4d4eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63a91deb00da41c0940784fe4e53c0f3 in term 0.
I20260925 16:36:55.326038 18612 leader_election.cc:302] T ccb9ec462b514d528ebbad653913963c P 63a91deb00da41c0940784fe4e53c0f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63a91deb00da41c0940784fe4e53c0f3, ad48a7044c2d422a83b37ed98ea9ab93; no voters:
I20260925 16:36:55.326341 19015 raft_consensus.cc:2781] T ccb9ec462b514d528ebbad653913963c P 63a91deb00da41c0940784fe4e53c0f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:36:55.326422 19015 raft_consensus.cc:482] T ccb9ec462b514d528ebbad653913963c P 63a91deb00da41c0940784fe4e53c0f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:36:55.326453 19015 raft_consensus.cc:3037] T ccb9ec462b514d528ebbad653913963c P 63a91deb00da41c0940784fe4e53c0f3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:36:55.327500 19015 raft_consensus.cc:504] T ccb9ec462b514d528ebbad653913963c P 63a91deb00da41c0940784fe4e53c0f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b8b0a91ae946b58421951383b4d4eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33815 } } peers { permanent_uuid: "63a91deb00da41c0940784fe4e53c0f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39935 } } peers { permanent_uuid: "ad48a7044c2d422a83b37ed98ea9ab93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40789 } }
I20260925 16:36:55.327888 19015 leader_election.cc:288] T ccb9ec462b514d528ebbad653913963c P 63a91deb00da41c0940784fe4e53c0f3 [CANDIDATE]: Term 1 election: Requested vote from peers 09b8b0a91ae946b58421951383b4d4eb (127.0.98.2:33815), ad48a7044c2d422a83b37ed98ea9ab93 (127.0.98.3:40789)
I20260925 16:36:55.328230 18810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccb9ec462b514d528ebbad653913963c" candidate_uuid: "63a91deb00da41c0940784fe4e53c0f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09b8b0a91ae946b58421951383b4d4eb"
I20260925 16:36:55.328289 18944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccb9ec462b514d528ebbad653913963c" candidate_uuid: "63a91deb00da41c0940784fe4e53c0f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad48a7044c2d422a83b37ed98ea9ab93"
I20260925 16:36:55.328347 18810 raft_consensus.cc:3037] T ccb9ec462b514d528ebbad653913963c P 09b8b0a91ae946b58421951383b4d4eb [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:36:55.328430 18944 raft_consensus.cc:3037] T ccb9ec462b514d528ebbad653913963c P ad48a7044c2d422a83b37ed98ea9ab93 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:36:55.329377 18810 raft_consensus.cc:2454] T ccb9ec462b514d528ebbad653913963c P 09b8b0a91ae946b58421951383b4d4eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63a91deb00da41c0940784fe4e53c0f3 in term 1.
I20260925 16:36:55.329502 18944 raft_consensus.cc:2454] T ccb9ec462b514d528ebbad653913963c P ad48a7044c2d422a83b37ed98ea9ab93 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63a91deb00da41c0940784fe4e53c0f3 in term 1.
I20260925 16:36:55.329674 18611 leader_election.cc:302] T ccb9ec462b514d528ebbad653913963c P 63a91deb00da41c0940784fe4e53c0f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09b8b0a91ae946b58421951383b4d4eb, 63a91deb00da41c0940784fe4e53c0f3; no voters:
I20260925 16:36:55.329844 19015 raft_consensus.cc:2781] T ccb9ec462b514d528ebbad653913963c P 63a91deb00da41c0940784fe4e53c0f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:36:55.330086 19015 raft_consensus.cc:686] T ccb9ec462b514d528ebbad653913963c P 63a91deb00da41c0940784fe4e53c0f3 [term 1 LEADER]: Becoming Leader. State: Replica: 63a91deb00da41c0940784fe4e53c0f3, State: Running, Role: LEADER
I20260925 16:36:55.330418 19015 consensus_queue.cc:237] T ccb9ec462b514d528ebbad653913963c P 63a91deb00da41c0940784fe4e53c0f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b8b0a91ae946b58421951383b4d4eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33815 } } peers { permanent_uuid: "63a91deb00da41c0940784fe4e53c0f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39935 } } peers { permanent_uuid: "ad48a7044c2d422a83b37ed98ea9ab93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40789 } }
I20260925 16:36:55.333770 18531 catalog_manager.cc:5495] T ccb9ec462b514d528ebbad653913963c P 63a91deb00da41c0940784fe4e53c0f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63a91deb00da41c0940784fe4e53c0f3 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "63a91deb00da41c0940784fe4e53c0f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b8b0a91ae946b58421951383b4d4eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63a91deb00da41c0940784fe4e53c0f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad48a7044c2d422a83b37ed98ea9ab93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40789 } health_report { overall_health: UNKNOWN } } }
I20260925 16:36:55.349771 19003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:55.346029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41706 (local address 127.0.98.1:39935)
0925 16:36:55.346176 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:55.346180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:55.346195 (+ 15us) server_negotiation.cc:407] Connection header received
0925 16:36:55.346354 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:55.346356 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:55.346413 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:55.346524 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:36:55.346818 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:55.347321 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:55.348193 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:55.348419 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:55.348525 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:55.348851 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:36:55.349326 (+ 475us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:36:55.349495 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:55.349584 (+ 89us) server_negotiation.cc:299] Negotiation successful
0925 16:36:55.349622 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 16:36:55.355897 18810 raft_consensus.cc:1261] T ccb9ec462b514d528ebbad653913963c P 09b8b0a91ae946b58421951383b4d4eb [term 1 FOLLOWER]: Refusing update from remote peer 63a91deb00da41c0940784fe4e53c0f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:36:55.355899 18944 raft_consensus.cc:1261] T ccb9ec462b514d528ebbad653913963c P ad48a7044c2d422a83b37ed98ea9ab93 [term 1 FOLLOWER]: Refusing update from remote peer 63a91deb00da41c0940784fe4e53c0f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:36:55.356573 19015 consensus_queue.cc:1035] T ccb9ec462b514d528ebbad653913963c P 63a91deb00da41c0940784fe4e53c0f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09b8b0a91ae946b58421951383b4d4eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:36:55.356782 19018 consensus_queue.cc:1035] T ccb9ec462b514d528ebbad653913963c P 63a91deb00da41c0940784fe4e53c0f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad48a7044c2d422a83b37ed98ea9ab93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:36:55.364941 19023 mvcc.cc:205] Tried to move back new op lower bound from 7333290866090315776 to 7333290865996009472. Current Snapshot: MvccSnapshot[applied={T|T < 7333290866090315776}]
I20260925 16:36:55.373590 18529 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42340:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:36:55.373749 18529 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42340:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:36:55.376261 18529 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c860c6cccc314cb798aff469c10e0ec1: Sending DeleteTablet for 3 replicas of tablet ccb9ec462b514d528ebbad653913963c
I20260925 16:36:55.377120 18924 tablet_service.cc:1514] Processing DeleteTablet for tablet ccb9ec462b514d528ebbad653913963c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:36:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60658
I20260925 16:36:55.377107 18656 tablet_service.cc:1514] Processing DeleteTablet for tablet ccb9ec462b514d528ebbad653913963c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:36:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41690
I20260925 16:36:55.377635 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 18593
I20260925 16:36:55.377772 19031 tablet_replica.cc:330] T ccb9ec462b514d528ebbad653913963c P ad48a7044c2d422a83b37ed98ea9ab93: stopping tablet replica
I20260925 16:36:55.377807 18790 tablet_service.cc:1514] Processing DeleteTablet for tablet ccb9ec462b514d528ebbad653913963c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:36:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45664
I20260925 16:36:55.378157 19031 raft_consensus.cc:2229] T ccb9ec462b514d528ebbad653913963c P ad48a7044c2d422a83b37ed98ea9ab93 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:36:55.378490 19031 raft_consensus.cc:2258] T ccb9ec462b514d528ebbad653913963c P ad48a7044c2d422a83b37ed98ea9ab93 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:36:55.380024 19031 ts_tablet_manager.cc:1905] T ccb9ec462b514d528ebbad653913963c P ad48a7044c2d422a83b37ed98ea9ab93: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:36:55.382269 19031 ts_tablet_manager.cc:1918] T ccb9ec462b514d528ebbad653913963c P ad48a7044c2d422a83b37ed98ea9ab93: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:36:55.382350 19031 log.cc:1196] T ccb9ec462b514d528ebbad653913963c P ad48a7044c2d422a83b37ed98ea9ab93: Deleting WAL directory at /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353966814071-392-0/minicluster-data/ts-2/wal/wals/ccb9ec462b514d528ebbad653913963c
I20260925 16:36:55.382714 19033 tablet_replica.cc:330] T ccb9ec462b514d528ebbad653913963c P 09b8b0a91ae946b58421951383b4d4eb: stopping tablet replica
I20260925 16:36:55.382742 19031 ts_tablet_manager.cc:1939] T ccb9ec462b514d528ebbad653913963c P ad48a7044c2d422a83b37ed98ea9ab93: Deleting consensus metadata
I20260925 16:36:55.383085 19033 raft_consensus.cc:2229] T ccb9ec462b514d528ebbad653913963c P 09b8b0a91ae946b58421951383b4d4eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:36:55.383409 19033 raft_consensus.cc:2258] T ccb9ec462b514d528ebbad653913963c P 09b8b0a91ae946b58421951383b4d4eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:36:55.384025 18518 catalog_manager.cc:4841] TS ad48a7044c2d422a83b37ed98ea9ab93 (127.0.98.3:40789): tablet ccb9ec462b514d528ebbad653913963c (table test-table [id=61c32c95684841fbaa6f86b4cc594e53]) successfully deleted
I20260925 16:36:55.384883 19033 ts_tablet_manager.cc:1905] T ccb9ec462b514d528ebbad653913963c P 09b8b0a91ae946b58421951383b4d4eb: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 16:36:55.385123 18516 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39935 (error 108)
I20260925 16:36:55.385993 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 18727
I20260925 16:36:55.386205 19002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:55.385869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39935 (local address 127.0.0.1:40772)
0925 16:36:55.386014 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:55.386095 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260925 16:36:55.387985 18516 catalog_manager.cc:4584] TS 63a91deb00da41c0940784fe4e53c0f3 (127.0.98.1:39935): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ccb9ec462b514d528ebbad653913963c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39935: connect: Connection refused (error 111)
I20260925 16:36:55.392287 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 18861
I20260925 16:36:55.393997 19002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:55.392510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33815 (local address 127.0.0.1:54292)
0925 16:36:55.393861 (+ 1351us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:55.393926 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:33815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1236}
I20260925 16:36:55.397709 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 18499
2026-09-25T16:36:55Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2332 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-25T16:36:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:36:55Z Disabled control of system clock
I20260925 16:36:55.427716 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353966814071-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353966814071-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353966814071-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33699
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33699
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 16:36:55.536114 19038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:36:55.536375 19038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:36:55.536410 19038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:36:55.540315 19038 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:36:55.540370 19038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:36:55.540385 19038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:36:55.540401 19038 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:36:55.540416 19038 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:36:55.544989 19038 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353966814071-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353966814071-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33699
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353966814071-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:36:55.545986 19038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:36:55.547302 19038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:36:55.555763 19043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:36:55.555773 19046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:36:55.555824 19038 server_base.cc:1034] running on GCE node
W20260925 16:36:55.555770 19044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:36:55.556978 19038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:36:55.558066 19038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:36:55.559263 19038 hybrid_clock.cc:648] HybridClock initialized: now 1790354215559247 us; error 27 us; skew 500 ppm
I20260925 16:36:55.561203 19038 webserver.cc:460] Webserver started at http://127.0.98.62:41279/ using document root <none> and password file <none>
I20260925 16:36:55.561825 19038 fs_manager.cc:359] Metadata directory not provided
I20260925 16:36:55.561923 19038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:36:55.562181 19038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:36:55.564325 19038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353966814071-392-0/minicluster-data/master-0/data/instance:
uuid: "4485c0757fc545d9b18a39519c461fec"
format_stamp: "Formatted at 2026-09-25 16:36:55 on dist-test-slave-65pf"
server_key: "ed0beaef80c887bf89341455d7e394fa"
server_key_iv: "7fd440f645f73f1fc7e4312e53fee737"
server_key_version: "encryptionkey@0"
I20260925 16:36:55.564905 19038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353966814071-392-0/minicluster-data/master-0/wal/instance:
uuid: "4485c0757fc545d9b18a39519c461fec"
format_stamp: "Formatted at 2026-09-25 16:36:55 on dist-test-slave-65pf"
server_key: "ed0beaef80c887bf89341455d7e394fa"
server_key_iv: "7fd440f645f73f1fc7e4312e53fee737"
server_key_version: "encryptionkey@0"
I20260925 16:36:55.568554 19038 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 16:36:55.570892 19052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:55.572078 19038 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 16:36:55.572220 19038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353966814071-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353966814071-392-0/minicluster-data/master-0/wal
uuid: "4485c0757fc545d9b18a39519c461fec"
format_stamp: "Formatted at 2026-09-25 16:36:55 on dist-test-slave-65pf"
server_key: "ed0beaef80c887bf89341455d7e394fa"
server_key_iv: "7fd440f645f73f1fc7e4312e53fee737"
server_key_version: "encryptionkey@0"
I20260925 16:36:55.572345 19038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353966814071-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353966814071-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353966814071-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:36:55.595415 19038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:36:55.596164 19038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:36:55.596380 19038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:36:55.603199 19038 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33699
I20260925 16:36:55.603225 19104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33699 every 8 connection(s)
I20260925 16:36:55.604117 19038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:36:55.607219 19105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:36:55.612612 19105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4485c0757fc545d9b18a39519c461fec: Bootstrap starting.
I20260925 16:36:55.613468 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 19038
I20260925 16:36:55.613600 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353966814071-392-0/minicluster-data/master-0/wal/instance
I20260925 16:36:55.613951 392 external_mini_cluster.cc:1428] Setting key c721c0c5aae2ad95a31e3e7ffdc9bed0
I20260925 16:36:55.615762 19105 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4485c0757fc545d9b18a39519c461fec: Neither blocks nor log segments found. Creating new log.
I20260925 16:36:55.616487 19105 log.cc:824] T 00000000000000000000000000000000 P 4485c0757fc545d9b18a39519c461fec: Log is configured to *not* fsync() on all Append() calls
I20260925 16:36:55.618527 19105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4485c0757fc545d9b18a39519c461fec: No bootstrap required, opened a new log
I20260925 16:36:55.622896 19105 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4485c0757fc545d9b18a39519c461fec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4485c0757fc545d9b18a39519c461fec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33699 } }
I20260925 16:36:55.623114 19105 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4485c0757fc545d9b18a39519c461fec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:36:55.623170 19105 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4485c0757fc545d9b18a39519c461fec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4485c0757fc545d9b18a39519c461fec, State: Initialized, Role: FOLLOWER
I20260925 16:36:55.623435 19108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:55.615152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36874 (local address 127.0.98.62:33699)
0925 16:36:55.615921 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:55.615938 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:55.615973 (+ 35us) server_negotiation.cc:407] Connection header received
0925 16:36:55.616632 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:55.616660 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:55.617162 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:55.617500 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:36:55.617836 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:55.618872 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:55.620040 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:55.620525 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:55.620638 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:55.621495 (+ 857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:36:55.621500 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:36:55.621514 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:36:55.621907 (+ 393us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:36:55.621961 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:36:55.621980 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:36:55.622057 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:55.622231 (+ 174us) server_negotiation.cc:299] Negotiation successful
0925 16:36:55.622536 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":159,"threads_started":1}
I20260925 16:36:55.623711 19105 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4485c0757fc545d9b18a39519c461fec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4485c0757fc545d9b18a39519c461fec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33699 } }
I20260925 16:36:55.623862 19105 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4485c0757fc545d9b18a39519c461fec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:36:55.623929 19105 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4485c0757fc545d9b18a39519c461fec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:36:55.624028 19105 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4485c0757fc545d9b18a39519c461fec [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:36:55.625116 19105 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4485c0757fc545d9b18a39519c461fec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4485c0757fc545d9b18a39519c461fec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33699 } }
I20260925 16:36:55.625484 19105 leader_election.cc:302] T 00000000000000000000000000000000 P 4485c0757fc545d9b18a39519c461fec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4485c0757fc545d9b18a39519c461fec; no voters:
I20260925 16:36:55.625825 19105 leader_election.cc:288] T 00000000000000000000000000000000 P 4485c0757fc545d9b18a39519c461fec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:36:55.625898 19110 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4485c0757fc545d9b18a39519c461fec [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:36:55.626089 19110 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4485c0757fc545d9b18a39519c461fec [term 1 LEADER]: Becoming Leader. State: Replica: 4485c0757fc545d9b18a39519c461fec, State: Running, Role: LEADER
I20260925 16:36:55.626391 19110 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4485c0757fc545d9b18a39519c461fec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4485c0757fc545d9b18a39519c461fec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33699 } }
I20260925 16:36:55.627112 19105 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4485c0757fc545d9b18a39519c461fec [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:36:55.628153 19112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4485c0757fc545d9b18a39519c461fec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4485c0757fc545d9b18a39519c461fec. Latest consensus state: current_term: 1 leader_uuid: "4485c0757fc545d9b18a39519c461fec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4485c0757fc545d9b18a39519c461fec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33699 } } }
I20260925 16:36:55.628295 19112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4485c0757fc545d9b18a39519c461fec [sys.catalog]: This master's current role is: LEADER
I20260925 16:36:55.628157 19111 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4485c0757fc545d9b18a39519c461fec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4485c0757fc545d9b18a39519c461fec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4485c0757fc545d9b18a39519c461fec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33699 } } }
I20260925 16:36:55.628465 19111 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4485c0757fc545d9b18a39519c461fec [sys.catalog]: This master's current role is: LEADER
W20260925 16:36:55.632324 19126 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4485c0757fc545d9b18a39519c461fec: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 16:36:55.632406 19126 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 16:36:55.632473 19119 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:36:55.633107 19119 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:36:55.638082 19119 catalog_manager.cc:1347] Generated new cluster ID: 737e27cef785415a9f7298e64fab3e5f
I20260925 16:36:55.638152 19119 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:36:55.660552 19119 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:36:55.661589 19119 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:36:55.670934 19119 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4485c0757fc545d9b18a39519c461fec: Generated new TSK 0
I20260925 16:36:55.671622 19119 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:36:55.679481 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353966814071-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353966814071-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353966814071-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33699
--builtin_ntp_servers=127.0.98.20:38783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 16:36:55.791050 19129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:36:55.791297 19129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:36:55.791337 19129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:36:55.795333 19129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:36:55.795384 19129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:36:55.795439 19129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 16:36:55.800078 19129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353966814071-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353966814071-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353966814071-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:36:55.801137 19129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:36:55.802503 19129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:36:55.811865 19137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:36:55.811831 19134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:36:55.811834 19135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:36:55.812191 19129 server_base.cc:1034] running on GCE node
I20260925 16:36:55.812527 19129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:36:55.813128 19129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:36:55.814334 19129 hybrid_clock.cc:648] HybridClock initialized: now 1790354215814318 us; error 31 us; skew 500 ppm
I20260925 16:36:55.816339 19129 webserver.cc:460] Webserver started at http://127.0.98.1:44875/ using document root <none> and password file <none>
I20260925 16:36:55.816923 19129 fs_manager.cc:359] Metadata directory not provided
I20260925 16:36:55.817044 19129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:36:55.817294 19129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:36:55.819495 19129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353966814071-392-0/minicluster-data/ts-0/data/instance:
uuid: "d6b54f5b8c494256a21c2fb9ba83c7c5"
format_stamp: "Formatted at 2026-09-25 16:36:55 on dist-test-slave-65pf"
server_key: "71e0dfe12dfbfabe2103e83209f9fbb4"
server_key_iv: "733b83af587cd070d0a19b9868fa890a"
server_key_version: "encryptionkey@0"
I20260925 16:36:55.820088 19129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353966814071-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d6b54f5b8c494256a21c2fb9ba83c7c5"
format_stamp: "Formatted at 2026-09-25 16:36:55 on dist-test-slave-65pf"
server_key: "71e0dfe12dfbfabe2103e83209f9fbb4"
server_key_iv: "733b83af587cd070d0a19b9868fa890a"
server_key_version: "encryptionkey@0"
I20260925 16:36:55.823706 19129 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 16:36:55.826122 19143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:55.827387 19129 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:36:55.827538 19129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353966814071-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353966814071-392-0/minicluster-data/ts-0/wal
uuid: "d6b54f5b8c494256a21c2fb9ba83c7c5"
format_stamp: "Formatted at 2026-09-25 16:36:55 on dist-test-slave-65pf"
server_key: "71e0dfe12dfbfabe2103e83209f9fbb4"
server_key_iv: "733b83af587cd070d0a19b9868fa890a"
server_key_version: "encryptionkey@0"
I20260925 16:36:55.827658 19129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353966814071-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353966814071-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353966814071-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:36:55.847095 19129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:36:55.847828 19129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:36:55.848026 19129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:36:55.848599 19129 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:36:55.849531 19129 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:36:55.849604 19129 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:55.849674 19129 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:36:55.849735 19129 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:55.858039 19129 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46007
I20260925 16:36:55.858111 19256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46007 every 8 connection(s)
I20260925 16:36:55.859062 19129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
I20260925 16:36:55.865808 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 19129
I20260925 16:36:55.865940 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353966814071-392-0/minicluster-data/ts-0/wal/instance
I20260925 16:36:55.866256 392 external_mini_cluster.cc:1428] Setting key 5bcaf5cb07d1d0940b29c21823d3d19e
I20260925 16:36:55.868443 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353966814071-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353966814071-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353966814071-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33699
--builtin_ntp_servers=127.0.98.20:38783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 16:36:55.868613 19108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:55.860829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41157 (local address 127.0.98.62:33699)
0925 16:36:55.860962 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:55.860966 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:55.861977 (+ 1011us) server_negotiation.cc:407] Connection header received
0925 16:36:55.863345 (+ 1368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:55.863348 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:55.863405 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:55.863496 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:36:55.864368 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:55.864924 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:55.866388 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:55.866666 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:55.866788 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:55.868069 (+ 1281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:36:55.868073 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:36:55.868075 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:36:55.868088 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:36:55.868113 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:36:55.868126 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:36:55.868205 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:55.868373 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 16:36:55.868437 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 16:36:55.869431 19259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:55.861214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33699 (local address 127.0.98.1:41157)
0925 16:36:55.861855 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:55.861894 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 16:36:55.863144 (+ 1250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:36:55.863653 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:36:55.863659 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:36:55.863737 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:36:55.864241 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:55.864253 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:55.865050 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:55.865053 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:55.866248 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:55.866259 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:55.866832 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:55.866836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:55.867042 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:55.867796 (+ 754us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 16:36:55.867798 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:36:55.867847 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 16:36:55.867910 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:36:55.868228 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:36:55.868236 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:36:55.868240 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 16:36:55.868336 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 16:36:55.868600 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":109,"threads_started":1}
I20260925 16:36:55.870662 19257 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33699
I20260925 16:36:55.870906 19257 heartbeater.cc:464] Registering TS with master...
I20260925 16:36:55.871429 19257 heartbeater.cc:511] Master 127.0.98.62:33699 requested a full tablet report, sending...
I20260925 16:36:55.872861 19069 ts_manager.cc:194] Registered new tserver with Master: d6b54f5b8c494256a21c2fb9ba83c7c5 (127.0.98.1:46007)
I20260925 16:36:55.873966 19069 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:41157
W20260925 16:36:55.976931 19260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:36:55.977221 19260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:36:55.977288 19260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:36:55.981243 19260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:36:55.981317 19260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:36:55.981403 19260 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 16:36:55.986017 19260 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353966814071-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353966814071-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19260
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353966814071-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:36:55.987078 19260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:36:55.988404 19260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:36:55.997637 19266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:36:55.997664 19265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:36:55.997649 19268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:36:55.997941 19260 server_base.cc:1034] running on GCE node
I20260925 16:36:55.998404 19260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:36:55.999109 19260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:36:56.000301 19260 hybrid_clock.cc:648] HybridClock initialized: now 1790354216000297 us; error 31 us; skew 500 ppm
I20260925 16:36:56.001977 19260 webserver.cc:460] Webserver started at http://127.0.98.2:37397/ using document root <none> and password file <none>
I20260925 16:36:56.002544 19260 fs_manager.cc:359] Metadata directory not provided
I20260925 16:36:56.002660 19260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:36:56.002877 19260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:36:56.004834 19260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353966814071-392-0/minicluster-data/ts-1/data/instance:
uuid: "f36fb3d21c8a45f9b84ec13420eb8978"
format_stamp: "Formatted at 2026-09-25 16:36:55 on dist-test-slave-65pf"
server_key: "42242c47a09998fe2ffc0469d1a436a9"
server_key_iv: "b10947050caf43852b30fde53033000e"
server_key_version: "encryptionkey@0"
I20260925 16:36:56.005381 19260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353966814071-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f36fb3d21c8a45f9b84ec13420eb8978"
format_stamp: "Formatted at 2026-09-25 16:36:55 on dist-test-slave-65pf"
server_key: "42242c47a09998fe2ffc0469d1a436a9"
server_key_iv: "b10947050caf43852b30fde53033000e"
server_key_version: "encryptionkey@0"
I20260925 16:36:56.008946 19260 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260925 16:36:56.011237 19274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:56.012362 19260 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260925 16:36:56.012496 19260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353966814071-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353966814071-392-0/minicluster-data/ts-1/wal
uuid: "f36fb3d21c8a45f9b84ec13420eb8978"
format_stamp: "Formatted at 2026-09-25 16:36:55 on dist-test-slave-65pf"
server_key: "42242c47a09998fe2ffc0469d1a436a9"
server_key_iv: "b10947050caf43852b30fde53033000e"
server_key_version: "encryptionkey@0"
I20260925 16:36:56.012601 19260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353966814071-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353966814071-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353966814071-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:36:56.032097 19260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:36:56.032756 19260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:36:56.032955 19260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:36:56.033522 19260 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:36:56.034425 19260 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:36:56.034495 19260 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:56.034564 19260 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:36:56.034636 19260 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:56.043341 19260 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41153
I20260925 16:36:56.043362 19387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41153 every 8 connection(s)
I20260925 16:36:56.044275 19260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
I20260925 16:36:56.044773 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 19260
I20260925 16:36:56.044924 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353966814071-392-0/minicluster-data/ts-1/wal/instance
I20260925 16:36:56.045276 392 external_mini_cluster.cc:1428] Setting key 680e066d8ab3b2d405d62e43fb8e1c83
I20260925 16:36:56.047403 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353966814071-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353966814071-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353966814071-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33699
--builtin_ntp_servers=127.0.98.20:38783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 16:36:56.055377 19108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:56.046189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56981 (local address 127.0.98.62:33699)
0925 16:36:56.046327 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:56.046332 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:56.047639 (+ 1307us) server_negotiation.cc:407] Connection header received
0925 16:36:56.049333 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:56.049337 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:56.049382 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:56.049452 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:36:56.050701 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:56.051291 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:56.053189 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:56.053398 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:56.053469 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:56.054904 (+ 1435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:36:56.054907 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:36:56.054909 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:36:56.054922 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:36:56.054938 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:36:56.054946 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:36:56.055004 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:56.055205 (+ 201us) server_negotiation.cc:299] Negotiation successful
0925 16:36:56.055256 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 16:36:56.056733 19390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:56.046738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33699 (local address 127.0.98.2:56981)
0925 16:36:56.047494 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:56.047546 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 16:36:56.049092 (+ 1546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:36:56.049649 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:36:56.049659 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:36:56.049783 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:36:56.050475 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:56.050497 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:56.051437 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:56.051444 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:56.053070 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:56.053083 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:56.053569 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:56.053575 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:56.053849 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:56.054556 (+ 707us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 16:36:56.054560 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:36:56.054644 (+ 84us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 16:36:56.054747 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:36:56.055036 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:36:56.055048 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:36:56.055056 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 16:36:56.055189 (+ 133us) client_negotiation.cc:240] Negotiation successful
0925 16:36:56.055607 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":147,"threads_started":1}
I20260925 16:36:56.057947 19388 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33699
I20260925 16:36:56.058173 19388 heartbeater.cc:464] Registering TS with master...
I20260925 16:36:56.058722 19388 heartbeater.cc:511] Master 127.0.98.62:33699 requested a full tablet report, sending...
I20260925 16:36:56.059715 19069 ts_manager.cc:194] Registered new tserver with Master: f36fb3d21c8a45f9b84ec13420eb8978 (127.0.98.2:41153)
I20260925 16:36:56.060268 19069 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:56981
W20260925 16:36:56.158191 19391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:36:56.158465 19391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:36:56.158537 19391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:36:56.162518 19391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:36:56.162621 19391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:36:56.162729 19391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 16:36:56.167456 19391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353966814071-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353966814071-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19391
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353966814071-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:36:56.168524 19391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:36:56.169883 19391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:36:56.179275 19397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:36:56.179374 19396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:36:56.179387 19399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:36:56.179615 19391 server_base.cc:1034] running on GCE node
I20260925 16:36:56.179976 19391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:36:56.180569 19391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:36:56.181730 19391 hybrid_clock.cc:648] HybridClock initialized: now 1790354216181690 us; error 50 us; skew 500 ppm
I20260925 16:36:56.183609 19391 webserver.cc:460] Webserver started at http://127.0.98.3:37291/ using document root <none> and password file <none>
I20260925 16:36:56.184173 19391 fs_manager.cc:359] Metadata directory not provided
I20260925 16:36:56.184232 19391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:36:56.184473 19391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:36:56.186489 19391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353966814071-392-0/minicluster-data/ts-2/data/instance:
uuid: "d32fe0806fe84cf789df0e45b0e0e7ef"
format_stamp: "Formatted at 2026-09-25 16:36:56 on dist-test-slave-65pf"
server_key: "f72accfe40325cafedca8ea293f3683c"
server_key_iv: "67fe50f2f6d6781888ab5b264aee0c98"
server_key_version: "encryptionkey@0"
I20260925 16:36:56.187094 19391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353966814071-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d32fe0806fe84cf789df0e45b0e0e7ef"
format_stamp: "Formatted at 2026-09-25 16:36:56 on dist-test-slave-65pf"
server_key: "f72accfe40325cafedca8ea293f3683c"
server_key_iv: "67fe50f2f6d6781888ab5b264aee0c98"
server_key_version: "encryptionkey@0"
I20260925 16:36:56.190485 19391 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 16:36:56.192740 19405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:56.194053 19391 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:36:56.194175 19391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353966814071-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353966814071-392-0/minicluster-data/ts-2/wal
uuid: "d32fe0806fe84cf789df0e45b0e0e7ef"
format_stamp: "Formatted at 2026-09-25 16:36:56 on dist-test-slave-65pf"
server_key: "f72accfe40325cafedca8ea293f3683c"
server_key_iv: "67fe50f2f6d6781888ab5b264aee0c98"
server_key_version: "encryptionkey@0"
I20260925 16:36:56.194264 19391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353966814071-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353966814071-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353966814071-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:36:56.221101 19391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:36:56.221738 19391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:36:56.221894 19391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:36:56.222421 19391 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:36:56.223368 19391 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:36:56.223415 19391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:56.223453 19391 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:36:56.223469 19391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:56.232560 19391 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41733
I20260925 16:36:56.232630 19518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41733 every 8 connection(s)
I20260925 16:36:56.233493 19391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
I20260925 16:36:56.234153 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 19391
I20260925 16:36:56.234295 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353966814071-392-0/minicluster-data/ts-2/wal/instance
I20260925 16:36:56.234565 392 external_mini_cluster.cc:1428] Setting key dd00e6d46a187685c7e0a488b9d94216
I20260925 16:36:56.242494 19108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:56.236952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42483 (local address 127.0.98.62:33699)
0925 16:36:56.237080 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:56.237084 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:56.237095 (+ 11us) server_negotiation.cc:407] Connection header received
0925 16:36:56.238310 (+ 1215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:56.238313 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:56.238352 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:56.238417 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:36:56.239294 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:56.239805 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:56.240846 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:56.241049 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:56.241118 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:56.242139 (+ 1021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:36:56.242142 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:36:56.242144 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:36:56.242153 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:36:56.242168 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:36:56.242174 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:36:56.242226 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:56.242346 (+ 120us) server_negotiation.cc:299] Negotiation successful
0925 16:36:56.242389 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 16:36:56.243294 19521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:56.236086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33699 (local address 127.0.98.3:42483)
0925 16:36:56.236844 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:56.236903 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 16:36:56.238140 (+ 1237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:36:56.238548 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:36:56.238554 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:36:56.238700 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:36:56.239181 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:56.239193 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:56.239908 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:56.239910 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:56.240747 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:56.240755 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:56.241166 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:56.241169 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:56.241303 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:56.241923 (+ 620us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 16:36:56.241924 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:36:56.241961 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 16:36:56.242020 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:36:56.242240 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:36:56.242244 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:36:56.242248 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 16:36:56.242333 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 16:36:56.242529 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":105,"threads_started":1}
I20260925 16:36:56.244342 19519 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33699
I20260925 16:36:56.244558 19519 heartbeater.cc:464] Registering TS with master...
I20260925 16:36:56.245028 19519 heartbeater.cc:511] Master 127.0.98.62:33699 requested a full tablet report, sending...
I20260925 16:36:56.245950 19069 ts_manager.cc:194] Registered new tserver with Master: d32fe0806fe84cf789df0e45b0e0e7ef (127.0.98.3:41733)
I20260925 16:36:56.246502 19069 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:42483
I20260925 16:36:56.248667 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 16:36:56.254125 19108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:56.250510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36876 (local address 127.0.98.62:33699)
0925 16:36:56.250663 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:56.250667 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:56.250837 (+ 170us) server_negotiation.cc:407] Connection header received
0925 16:36:56.251037 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:56.251040 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:56.251096 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:56.251161 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:36:56.251420 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:56.251908 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:56.252676 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:56.252900 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:56.252987 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:56.253700 (+ 713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:36:56.253703 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:36:56.253705 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:36:56.253715 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:36:56.253731 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:36:56.253738 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:36:56.253844 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:56.253933 (+ 89us) server_negotiation.cc:299] Negotiation successful
0925 16:36:56.253962 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 16:36:56.256807 19069 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36876:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 16:36:56.259749 19069 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:36:56.273279 19532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:56.267925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41153 (local address 127.0.0.1:39198)
0925 16:36:56.269002 (+ 1077us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:56.269016 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:36:56.269109 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:36:56.269576 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:36:56.269579 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:36:56.269596 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:36:56.269670 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:56.269677 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:56.270969 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:56.270977 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:56.271984 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:56.271992 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:56.272731 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:56.272734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:56.272840 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:56.272874 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 16:36:56.272986 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 16:36:56.273110 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1021,"mutex_wait_us":2,"thread_start_us":95,"threads_started":1}
I20260925 16:36:56.273288 19534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:56.268295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39198 (local address 127.0.98.2:41153)
0925 16:36:56.268867 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:56.268877 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:56.269194 (+ 317us) server_negotiation.cc:407] Connection header received
0925 16:36:56.269270 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:56.269277 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:56.269438 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:56.269586 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:36:56.269786 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:56.270858 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:56.272098 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:56.272610 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:56.272733 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:56.272910 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:56.273013 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 16:36:56.273104 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":108,"threads_started":1}
I20260925 16:36:56.273584 19533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:56.267732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41733 (local address 127.0.0.1:34980)
0925 16:36:56.268427 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:56.268440 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:36:56.268512 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:36:56.269323 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:36:56.269329 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:36:56.269359 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:36:56.269502 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:56.269512 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:56.270907 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:56.270911 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:56.271998 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:56.272006 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:56.273090 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:56.273094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:56.273258 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:56.273297 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 16:36:56.273370 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 16:36:56.273453 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"mutex_wait_us":73,"spinlock_wait_cycles":2304,"thread_start_us":93,"threads_started":1}
I20260925 16:36:56.273684 19531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:56.267709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48746 (local address 127.0.98.1:46007)
0925 16:36:56.268637 (+ 928us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:56.268689 (+ 52us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:56.268717 (+ 28us) server_negotiation.cc:407] Connection header received
0925 16:36:56.268791 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:56.268796 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:56.269017 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:56.269162 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:36:56.269628 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:56.270398 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:56.271927 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:56.272694 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:56.272796 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:56.272975 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:56.273445 (+ 470us) server_negotiation.cc:299] Negotiation successful
0925 16:36:56.273533 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":812,"thread_start_us":118,"threads_started":1}
I20260925 16:36:56.273902 19530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:56.267560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46007 (local address 127.0.0.1:48746)
0925 16:36:56.268169 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:56.268215 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 16:36:56.268334 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:36:56.269322 (+ 988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:36:56.269326 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:36:56.269361 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:36:56.269487 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:56.269492 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:56.270502 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:56.270508 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:56.271796 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:56.271805 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:56.273036 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:56.273040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:56.273237 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:56.273300 (+ 63us) client_negotiation.cc:769] Sending connection context
0925 16:36:56.273375 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 16:36:56.273674 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"mutex_wait_us":341,"spinlock_wait_cycles":8704,"thread_start_us":70,"threads_started":1}
I20260925 16:36:56.274322 19535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:56.268276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34980 (local address 127.0.98.3:41733)
0925 16:36:56.268706 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:56.268720 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:56.268743 (+ 23us) server_negotiation.cc:407] Connection header received
0925 16:36:56.268818 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:56.268823 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:56.269017 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:56.269163 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:36:56.269627 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:56.270793 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:56.272115 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:56.272808 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:56.272907 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:56.273043 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:56.274075 (+ 1032us) server_negotiation.cc:299] Negotiation successful
0925 16:36:56.274172 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":143,"threads_started":1}
I20260925 16:36:56.276461 19322 tablet_service.cc:1467] Processing CreateTablet for tablet cd22bc987e904400bdf7d71ef6c719b5 (DEFAULT_TABLE table=test-table [id=93ca13cec73e4eb497060a09b34f670e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:36:56.276530 19191 tablet_service.cc:1467] Processing CreateTablet for tablet cd22bc987e904400bdf7d71ef6c719b5 (DEFAULT_TABLE table=test-table [id=93ca13cec73e4eb497060a09b34f670e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:36:56.276532 19453 tablet_service.cc:1467] Processing CreateTablet for tablet cd22bc987e904400bdf7d71ef6c719b5 (DEFAULT_TABLE table=test-table [id=93ca13cec73e4eb497060a09b34f670e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:36:56.277438 19322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd22bc987e904400bdf7d71ef6c719b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:36:56.277493 19191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd22bc987e904400bdf7d71ef6c719b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:36:56.277493 19453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd22bc987e904400bdf7d71ef6c719b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:36:56.284291 19538 tablet_bootstrap.cc:492] T cd22bc987e904400bdf7d71ef6c719b5 P d6b54f5b8c494256a21c2fb9ba83c7c5: Bootstrap starting.
I20260925 16:36:56.284286 19537 tablet_bootstrap.cc:492] T cd22bc987e904400bdf7d71ef6c719b5 P d32fe0806fe84cf789df0e45b0e0e7ef: Bootstrap starting.
I20260925 16:36:56.285013 19536 tablet_bootstrap.cc:492] T cd22bc987e904400bdf7d71ef6c719b5 P f36fb3d21c8a45f9b84ec13420eb8978: Bootstrap starting.
I20260925 16:36:56.286633 19537 tablet_bootstrap.cc:654] T cd22bc987e904400bdf7d71ef6c719b5 P d32fe0806fe84cf789df0e45b0e0e7ef: Neither blocks nor log segments found. Creating new log.
I20260925 16:36:56.286633 19538 tablet_bootstrap.cc:654] T cd22bc987e904400bdf7d71ef6c719b5 P d6b54f5b8c494256a21c2fb9ba83c7c5: Neither blocks nor log segments found. Creating new log.
I20260925 16:36:56.286734 19536 tablet_bootstrap.cc:654] T cd22bc987e904400bdf7d71ef6c719b5 P f36fb3d21c8a45f9b84ec13420eb8978: Neither blocks nor log segments found. Creating new log.
I20260925 16:36:56.287412 19536 log.cc:824] T cd22bc987e904400bdf7d71ef6c719b5 P f36fb3d21c8a45f9b84ec13420eb8978: Log is configured to *not* fsync() on all Append() calls
I20260925 16:36:56.287472 19538 log.cc:824] T cd22bc987e904400bdf7d71ef6c719b5 P d6b54f5b8c494256a21c2fb9ba83c7c5: Log is configured to *not* fsync() on all Append() calls
I20260925 16:36:56.287472 19537 log.cc:824] T cd22bc987e904400bdf7d71ef6c719b5 P d32fe0806fe84cf789df0e45b0e0e7ef: Log is configured to *not* fsync() on all Append() calls
I20260925 16:36:56.289460 19538 tablet_bootstrap.cc:492] T cd22bc987e904400bdf7d71ef6c719b5 P d6b54f5b8c494256a21c2fb9ba83c7c5: No bootstrap required, opened a new log
I20260925 16:36:56.289460 19537 tablet_bootstrap.cc:492] T cd22bc987e904400bdf7d71ef6c719b5 P d32fe0806fe84cf789df0e45b0e0e7ef: No bootstrap required, opened a new log
I20260925 16:36:56.289474 19536 tablet_bootstrap.cc:492] T cd22bc987e904400bdf7d71ef6c719b5 P f36fb3d21c8a45f9b84ec13420eb8978: No bootstrap required, opened a new log
I20260925 16:36:56.289642 19536 ts_tablet_manager.cc:1397] T cd22bc987e904400bdf7d71ef6c719b5 P f36fb3d21c8a45f9b84ec13420eb8978: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260925 16:36:56.289646 19537 ts_tablet_manager.cc:1397] T cd22bc987e904400bdf7d71ef6c719b5 P d32fe0806fe84cf789df0e45b0e0e7ef: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260925 16:36:56.289675 19538 ts_tablet_manager.cc:1397] T cd22bc987e904400bdf7d71ef6c719b5 P d6b54f5b8c494256a21c2fb9ba83c7c5: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260925 16:36:56.292766 19536 raft_consensus.cc:348] T cd22bc987e904400bdf7d71ef6c719b5 P f36fb3d21c8a45f9b84ec13420eb8978 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b54f5b8c494256a21c2fb9ba83c7c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46007 } } peers { permanent_uuid: "d32fe0806fe84cf789df0e45b0e0e7ef" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41733 } } peers { permanent_uuid: "f36fb3d21c8a45f9b84ec13420eb8978" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41153 } }
I20260925 16:36:56.293004 19536 raft_consensus.cc:374] T cd22bc987e904400bdf7d71ef6c719b5 P f36fb3d21c8a45f9b84ec13420eb8978 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:36:56.293080 19536 raft_consensus.cc:729] T cd22bc987e904400bdf7d71ef6c719b5 P f36fb3d21c8a45f9b84ec13420eb8978 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f36fb3d21c8a45f9b84ec13420eb8978, State: Initialized, Role: FOLLOWER
I20260925 16:36:56.293527 19536 consensus_queue.cc:260] T cd22bc987e904400bdf7d71ef6c719b5 P f36fb3d21c8a45f9b84ec13420eb8978 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b54f5b8c494256a21c2fb9ba83c7c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46007 } } peers { permanent_uuid: "d32fe0806fe84cf789df0e45b0e0e7ef" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41733 } } peers { permanent_uuid: "f36fb3d21c8a45f9b84ec13420eb8978" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41153 } }
I20260925 16:36:56.293809 19537 raft_consensus.cc:348] T cd22bc987e904400bdf7d71ef6c719b5 P d32fe0806fe84cf789df0e45b0e0e7ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b54f5b8c494256a21c2fb9ba83c7c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46007 } } peers { permanent_uuid: "d32fe0806fe84cf789df0e45b0e0e7ef" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41733 } } peers { permanent_uuid: "f36fb3d21c8a45f9b84ec13420eb8978" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41153 } }
I20260925 16:36:56.294086 19537 raft_consensus.cc:374] T cd22bc987e904400bdf7d71ef6c719b5 P d32fe0806fe84cf789df0e45b0e0e7ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:36:56.293982 19538 raft_consensus.cc:348] T cd22bc987e904400bdf7d71ef6c719b5 P d6b54f5b8c494256a21c2fb9ba83c7c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b54f5b8c494256a21c2fb9ba83c7c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46007 } } peers { permanent_uuid: "d32fe0806fe84cf789df0e45b0e0e7ef" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41733 } } peers { permanent_uuid: "f36fb3d21c8a45f9b84ec13420eb8978" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41153 } }
I20260925 16:36:56.294139 19388 heartbeater.cc:503] Master 127.0.98.62:33699 was elected leader, sending a full tablet report...
I20260925 16:36:56.294157 19537 raft_consensus.cc:729] T cd22bc987e904400bdf7d71ef6c719b5 P d32fe0806fe84cf789df0e45b0e0e7ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d32fe0806fe84cf789df0e45b0e0e7ef, State: Initialized, Role: FOLLOWER
I20260925 16:36:56.294214 19538 raft_consensus.cc:374] T cd22bc987e904400bdf7d71ef6c719b5 P d6b54f5b8c494256a21c2fb9ba83c7c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:36:56.294283 19538 raft_consensus.cc:729] T cd22bc987e904400bdf7d71ef6c719b5 P d6b54f5b8c494256a21c2fb9ba83c7c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6b54f5b8c494256a21c2fb9ba83c7c5, State: Initialized, Role: FOLLOWER
I20260925 16:36:56.294488 19536 ts_tablet_manager.cc:1428] T cd22bc987e904400bdf7d71ef6c719b5 P f36fb3d21c8a45f9b84ec13420eb8978: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260925 16:36:56.294616 19537 consensus_queue.cc:260] T cd22bc987e904400bdf7d71ef6c719b5 P d32fe0806fe84cf789df0e45b0e0e7ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b54f5b8c494256a21c2fb9ba83c7c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46007 } } peers { permanent_uuid: "d32fe0806fe84cf789df0e45b0e0e7ef" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41733 } } peers { permanent_uuid: "f36fb3d21c8a45f9b84ec13420eb8978" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41153 } }
I20260925 16:36:56.294667 19538 consensus_queue.cc:260] T cd22bc987e904400bdf7d71ef6c719b5 P d6b54f5b8c494256a21c2fb9ba83c7c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b54f5b8c494256a21c2fb9ba83c7c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46007 } } peers { permanent_uuid: "d32fe0806fe84cf789df0e45b0e0e7ef" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41733 } } peers { permanent_uuid: "f36fb3d21c8a45f9b84ec13420eb8978" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41153 } }
I20260925 16:36:56.295336 19257 heartbeater.cc:503] Master 127.0.98.62:33699 was elected leader, sending a full tablet report...
I20260925 16:36:56.295615 19537 ts_tablet_manager.cc:1428] T cd22bc987e904400bdf7d71ef6c719b5 P d32fe0806fe84cf789df0e45b0e0e7ef: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 16:36:56.295615 19538 ts_tablet_manager.cc:1428] T cd22bc987e904400bdf7d71ef6c719b5 P d6b54f5b8c494256a21c2fb9ba83c7c5: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260925 16:36:56.295921 19519 heartbeater.cc:503] Master 127.0.98.62:33699 was elected leader, sending a full tablet report...
W20260925 16:36:56.296209 19389 tablet.cc:2369] T cd22bc987e904400bdf7d71ef6c719b5 P f36fb3d21c8a45f9b84ec13420eb8978: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:36:56.348268 19544 raft_consensus.cc:482] T cd22bc987e904400bdf7d71ef6c719b5 P d32fe0806fe84cf789df0e45b0e0e7ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:36:56.348460 19544 raft_consensus.cc:504] T cd22bc987e904400bdf7d71ef6c719b5 P d32fe0806fe84cf789df0e45b0e0e7ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b54f5b8c494256a21c2fb9ba83c7c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46007 } } peers { permanent_uuid: "d32fe0806fe84cf789df0e45b0e0e7ef" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41733 } } peers { permanent_uuid: "f36fb3d21c8a45f9b84ec13420eb8978" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41153 } }
I20260925 16:36:56.349649 19544 leader_election.cc:288] T cd22bc987e904400bdf7d71ef6c719b5 P d32fe0806fe84cf789df0e45b0e0e7ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6b54f5b8c494256a21c2fb9ba83c7c5 (127.0.98.1:46007), f36fb3d21c8a45f9b84ec13420eb8978 (127.0.98.2:41153)
I20260925 16:36:56.353463 19521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:56.349897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41153 (local address 127.0.98.3:52165)
0925 16:36:56.350116 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:56.350133 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 16:36:56.350263 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:36:56.350546 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:36:56.350550 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:36:56.350567 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:36:56.350710 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:56.350739 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:56.351600 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:56.351605 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:56.352375 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:56.352384 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:56.353055 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:56.353059 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:56.353163 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:56.353186 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 16:36:56.353256 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 16:36:56.353320 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147}
I20260925 16:36:56.353487 19534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:56.349996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52165 (local address 127.0.98.2:41153)
0925 16:36:56.350175 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:56.350182 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:56.350203 (+ 21us) server_negotiation.cc:407] Connection header received
0925 16:36:56.350384 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:56.350386 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:56.350443 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:56.350533 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:36:56.350846 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:56.351494 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:56.352500 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:56.352917 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:56.353036 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:56.353096 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:56.353287 (+ 191us) server_negotiation.cc:299] Negotiation successful
0925 16:36:56.353358 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 16:36:56.353632 19531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:56.349988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51597 (local address 127.0.98.1:46007)
0925 16:36:56.350115 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:56.350119 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:56.350510 (+ 391us) server_negotiation.cc:407] Connection header received
0925 16:36:56.350685 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:56.350688 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:56.350739 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:56.350822 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:36:56.351099 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:56.351623 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:56.352613 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:56.353073 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:56.353169 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:56.353222 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:56.353436 (+ 214us) server_negotiation.cc:299] Negotiation successful
0925 16:36:56.353513 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 16:36:56.354311 19342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd22bc987e904400bdf7d71ef6c719b5" candidate_uuid: "d32fe0806fe84cf789df0e45b0e0e7ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f36fb3d21c8a45f9b84ec13420eb8978" is_pre_election: true
I20260925 16:36:56.353643 19545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:56.349897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46007 (local address 127.0.98.3:51597)
0925 16:36:56.350402 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:56.350413 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 16:36:56.350532 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:36:56.350832 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:36:56.350835 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:36:56.350850 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:36:56.350974 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:56.350979 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:56.351735 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:56.351739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:56.352495 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:56.352503 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:56.353197 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:56.353200 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:56.353310 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:56.353330 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 16:36:56.353411 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 16:36:56.353469 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"mutex_wait_us":111,"thread_start_us":133,"threads_started":1}
I20260925 16:36:56.354704 19342 raft_consensus.cc:2454] T cd22bc987e904400bdf7d71ef6c719b5 P f36fb3d21c8a45f9b84ec13420eb8978 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d32fe0806fe84cf789df0e45b0e0e7ef in term 0.
I20260925 16:36:56.355242 19409 leader_election.cc:302] T cd22bc987e904400bdf7d71ef6c719b5 P d32fe0806fe84cf789df0e45b0e0e7ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d32fe0806fe84cf789df0e45b0e0e7ef, f36fb3d21c8a45f9b84ec13420eb8978; no voters:
I20260925 16:36:56.355278 19211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd22bc987e904400bdf7d71ef6c719b5" candidate_uuid: "d32fe0806fe84cf789df0e45b0e0e7ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6b54f5b8c494256a21c2fb9ba83c7c5" is_pre_election: true
I20260925 16:36:56.355552 19211 raft_consensus.cc:2454] T cd22bc987e904400bdf7d71ef6c719b5 P d6b54f5b8c494256a21c2fb9ba83c7c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d32fe0806fe84cf789df0e45b0e0e7ef in term 0.
I20260925 16:36:56.355572 19544 raft_consensus.cc:2781] T cd22bc987e904400bdf7d71ef6c719b5 P d32fe0806fe84cf789df0e45b0e0e7ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:36:56.355640 19544 raft_consensus.cc:482] T cd22bc987e904400bdf7d71ef6c719b5 P d32fe0806fe84cf789df0e45b0e0e7ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:36:56.355680 19544 raft_consensus.cc:3037] T cd22bc987e904400bdf7d71ef6c719b5 P d32fe0806fe84cf789df0e45b0e0e7ef [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:36:56.357129 19544 raft_consensus.cc:504] T cd22bc987e904400bdf7d71ef6c719b5 P d32fe0806fe84cf789df0e45b0e0e7ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b54f5b8c494256a21c2fb9ba83c7c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46007 } } peers { permanent_uuid: "d32fe0806fe84cf789df0e45b0e0e7ef" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41733 } } peers { permanent_uuid: "f36fb3d21c8a45f9b84ec13420eb8978" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41153 } }
I20260925 16:36:56.357563 19544 leader_election.cc:288] T cd22bc987e904400bdf7d71ef6c719b5 P d32fe0806fe84cf789df0e45b0e0e7ef [CANDIDATE]: Term 1 election: Requested vote from peers d6b54f5b8c494256a21c2fb9ba83c7c5 (127.0.98.1:46007), f36fb3d21c8a45f9b84ec13420eb8978 (127.0.98.2:41153)
I20260925 16:36:56.357954 19342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd22bc987e904400bdf7d71ef6c719b5" candidate_uuid: "d32fe0806fe84cf789df0e45b0e0e7ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f36fb3d21c8a45f9b84ec13420eb8978"
I20260925 16:36:56.357939 19211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd22bc987e904400bdf7d71ef6c719b5" candidate_uuid: "d32fe0806fe84cf789df0e45b0e0e7ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6b54f5b8c494256a21c2fb9ba83c7c5"
I20260925 16:36:56.358116 19342 raft_consensus.cc:3037] T cd22bc987e904400bdf7d71ef6c719b5 P f36fb3d21c8a45f9b84ec13420eb8978 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:36:56.358140 19211 raft_consensus.cc:3037] T cd22bc987e904400bdf7d71ef6c719b5 P d6b54f5b8c494256a21c2fb9ba83c7c5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:36:56.359505 19342 raft_consensus.cc:2454] T cd22bc987e904400bdf7d71ef6c719b5 P f36fb3d21c8a45f9b84ec13420eb8978 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d32fe0806fe84cf789df0e45b0e0e7ef in term 1.
I20260925 16:36:56.359591 19211 raft_consensus.cc:2454] T cd22bc987e904400bdf7d71ef6c719b5 P d6b54f5b8c494256a21c2fb9ba83c7c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d32fe0806fe84cf789df0e45b0e0e7ef in term 1.
I20260925 16:36:56.359844 19409 leader_election.cc:302] T cd22bc987e904400bdf7d71ef6c719b5 P d32fe0806fe84cf789df0e45b0e0e7ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d32fe0806fe84cf789df0e45b0e0e7ef, f36fb3d21c8a45f9b84ec13420eb8978; no voters:
I20260925 16:36:56.360080 19544 raft_consensus.cc:2781] T cd22bc987e904400bdf7d71ef6c719b5 P d32fe0806fe84cf789df0e45b0e0e7ef [term 1 FOLLOWER]: Leader election won for term 1
W20260925 16:36:56.360174 19258 tablet.cc:2369] T cd22bc987e904400bdf7d71ef6c719b5 P d6b54f5b8c494256a21c2fb9ba83c7c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:36:56.360316 19544 raft_consensus.cc:686] T cd22bc987e904400bdf7d71ef6c719b5 P d32fe0806fe84cf789df0e45b0e0e7ef [term 1 LEADER]: Becoming Leader. State: Replica: d32fe0806fe84cf789df0e45b0e0e7ef, State: Running, Role: LEADER
I20260925 16:36:56.360682 19544 consensus_queue.cc:237] T cd22bc987e904400bdf7d71ef6c719b5 P d32fe0806fe84cf789df0e45b0e0e7ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b54f5b8c494256a21c2fb9ba83c7c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46007 } } peers { permanent_uuid: "d32fe0806fe84cf789df0e45b0e0e7ef" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41733 } } peers { permanent_uuid: "f36fb3d21c8a45f9b84ec13420eb8978" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41153 } }
I20260925 16:36:56.363759 19069 catalog_manager.cc:5495] T cd22bc987e904400bdf7d71ef6c719b5 P d32fe0806fe84cf789df0e45b0e0e7ef reported cstate change: term changed from 0 to 1, leader changed from <none> to d32fe0806fe84cf789df0e45b0e0e7ef (127.0.98.3). New cstate: current_term: 1 leader_uuid: "d32fe0806fe84cf789df0e45b0e0e7ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b54f5b8c494256a21c2fb9ba83c7c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d32fe0806fe84cf789df0e45b0e0e7ef" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f36fb3d21c8a45f9b84ec13420eb8978" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41153 } health_report { overall_health: UNKNOWN } } }
I20260925 16:36:56.396122 19535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:56.392618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34996 (local address 127.0.98.3:41733)
0925 16:36:56.392744 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:56.392747 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:56.392762 (+ 15us) server_negotiation.cc:407] Connection header received
0925 16:36:56.392867 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:56.392869 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:56.392921 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:56.393024 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:36:56.393245 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:56.393740 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:56.394717 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:56.394938 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:56.395048 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:56.395330 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:36:56.395707 (+ 377us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:36:56.395880 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:56.395942 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 16:36:56.395972 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 16:36:56.401602 19342 raft_consensus.cc:1261] T cd22bc987e904400bdf7d71ef6c719b5 P f36fb3d21c8a45f9b84ec13420eb8978 [term 1 FOLLOWER]: Refusing update from remote peer d32fe0806fe84cf789df0e45b0e0e7ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:36:56.401604 19211 raft_consensus.cc:1261] T cd22bc987e904400bdf7d71ef6c719b5 P d6b54f5b8c494256a21c2fb9ba83c7c5 [term 1 FOLLOWER]: Refusing update from remote peer d32fe0806fe84cf789df0e45b0e0e7ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:36:56.402272 19544 consensus_queue.cc:1035] T cd22bc987e904400bdf7d71ef6c719b5 P d32fe0806fe84cf789df0e45b0e0e7ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "f36fb3d21c8a45f9b84ec13420eb8978" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:36:56.402464 19546 consensus_queue.cc:1035] T cd22bc987e904400bdf7d71ef6c719b5 P d32fe0806fe84cf789df0e45b0e0e7ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b54f5b8c494256a21c2fb9ba83c7c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:36:56.417697 19068 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:36:56.417884 19068 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:36:56.420102 19068 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4485c0757fc545d9b18a39519c461fec: Sending DeleteTablet for 3 replicas of tablet cd22bc987e904400bdf7d71ef6c719b5
I20260925 16:36:56.420833 19191 tablet_service.cc:1514] Processing DeleteTablet for tablet cd22bc987e904400bdf7d71ef6c719b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:36:56 UTC) from {username='slave'} at 127.0.0.1:48746
I20260925 16:36:56.420884 19453 tablet_service.cc:1514] Processing DeleteTablet for tablet cd22bc987e904400bdf7d71ef6c719b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:36:56 UTC) from {username='slave'} at 127.0.0.1:34980
I20260925 16:36:56.421080 19322 tablet_service.cc:1514] Processing DeleteTablet for tablet cd22bc987e904400bdf7d71ef6c719b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:36:56 UTC) from {username='slave'} at 127.0.0.1:39198
I20260925 16:36:56.421496 19561 tablet_replica.cc:330] T cd22bc987e904400bdf7d71ef6c719b5 P f36fb3d21c8a45f9b84ec13420eb8978: stopping tablet replica
I20260925 16:36:56.421502 19559 tablet_replica.cc:330] T cd22bc987e904400bdf7d71ef6c719b5 P d32fe0806fe84cf789df0e45b0e0e7ef: stopping tablet replica
I20260925 16:36:56.421793 19559 raft_consensus.cc:2229] T cd22bc987e904400bdf7d71ef6c719b5 P d32fe0806fe84cf789df0e45b0e0e7ef [term 1 LEADER]: Raft consensus shutting down.
I20260925 16:36:56.421825 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 19129
I20260925 16:36:56.422145 19559 raft_consensus.cc:2258] T cd22bc987e904400bdf7d71ef6c719b5 P d32fe0806fe84cf789df0e45b0e0e7ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:36:56.421813 19561 raft_consensus.cc:2229] T cd22bc987e904400bdf7d71ef6c719b5 P f36fb3d21c8a45f9b84ec13420eb8978 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:36:56.422727 19561 raft_consensus.cc:2258] T cd22bc987e904400bdf7d71ef6c719b5 P f36fb3d21c8a45f9b84ec13420eb8978 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:36:56.424118 19559 ts_tablet_manager.cc:1905] T cd22bc987e904400bdf7d71ef6c719b5 P d32fe0806fe84cf789df0e45b0e0e7ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:36:56.424120 19561 ts_tablet_manager.cc:1905] T cd22bc987e904400bdf7d71ef6c719b5 P f36fb3d21c8a45f9b84ec13420eb8978: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:36:56.426894 19559 ts_tablet_manager.cc:1918] T cd22bc987e904400bdf7d71ef6c719b5 P d32fe0806fe84cf789df0e45b0e0e7ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:36:56.427073 19559 log.cc:1196] T cd22bc987e904400bdf7d71ef6c719b5 P d32fe0806fe84cf789df0e45b0e0e7ef: Deleting WAL directory at /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353966814071-392-0/minicluster-data/ts-2/wal/wals/cd22bc987e904400bdf7d71ef6c719b5
I20260925 16:36:56.427174 19561 ts_tablet_manager.cc:1918] T cd22bc987e904400bdf7d71ef6c719b5 P f36fb3d21c8a45f9b84ec13420eb8978: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:36:56.427284 19561 log.cc:1196] T cd22bc987e904400bdf7d71ef6c719b5 P f36fb3d21c8a45f9b84ec13420eb8978: Deleting WAL directory at /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353966814071-392-0/minicluster-data/ts-1/wal/wals/cd22bc987e904400bdf7d71ef6c719b5
I20260925 16:36:56.427544 19559 ts_tablet_manager.cc:1939] T cd22bc987e904400bdf7d71ef6c719b5 P d32fe0806fe84cf789df0e45b0e0e7ef: Deleting consensus metadata
I20260925 16:36:56.427690 19561 ts_tablet_manager.cc:1939] T cd22bc987e904400bdf7d71ef6c719b5 P f36fb3d21c8a45f9b84ec13420eb8978: Deleting consensus metadata
I20260925 16:36:56.428910 19053 catalog_manager.cc:4841] TS d32fe0806fe84cf789df0e45b0e0e7ef (127.0.98.3:41733): tablet cd22bc987e904400bdf7d71ef6c719b5 (table test-table [id=93ca13cec73e4eb497060a09b34f670e]) successfully deleted
I20260925 16:36:56.429186 19056 catalog_manager.cc:4841] TS f36fb3d21c8a45f9b84ec13420eb8978 (127.0.98.2:41153): tablet cd22bc987e904400bdf7d71ef6c719b5 (table test-table [id=93ca13cec73e4eb497060a09b34f670e]) successfully deleted
W20260925 16:36:56.431689 19055 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46007 (error 108)
I20260925 16:36:56.432544 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 19260
I20260925 16:36:56.432780 19532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:56.432438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46007 (local address 127.0.0.1:48754)
0925 16:36:56.432578 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:56.432655 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 16:36:56.434757 19055 catalog_manager.cc:4584] TS d6b54f5b8c494256a21c2fb9ba83c7c5 (127.0.98.1:46007): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cd22bc987e904400bdf7d71ef6c719b5: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46007: connect: Connection refused (error 111)
I20260925 16:36:56.439146 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 19391
I20260925 16:36:56.444016 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 19038
2026-09-25T16:36:56Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1044 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:36:56 dist-test-slave-65pf krb5kdc[19565](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:36:56 dist-test-slave-65pf krb5kdc[19565](info): set up 2 sockets
Sep 25 16:36:56 dist-test-slave-65pf krb5kdc[19565](info): commencing operation
krb5kdc: starting...
W20260925 16:36:57.474727 392 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/krb5kdc/test-user.keytab.
Sep 25 16:36:57 dist-test-slave-65pf krb5kdc[19565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:36:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:36:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 16:36:57.594282 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46555
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46555
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:36:57.703619 19578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:36:57.703928 19578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:36:57.704010 19578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:36:57.708074 19578 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:36:57.708170 19578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:36:57.708209 19578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:36:57.708263 19578 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:36:57.708302 19578 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:36:57.713589 19578 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46555
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:36:57.714669 19578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:36:57.715700 19578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:36:57.724092 19583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:36:57.724082 19586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:36:57.724084 19584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:36:57.724464 19578 server_base.cc:1034] running on GCE node
I20260925 16:36:57.725208 19578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:36:57.726310 19578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:36:57.727501 19578 hybrid_clock.cc:648] HybridClock initialized: now 1790354217727488 us; error 37 us; skew 500 ppm
Sep 25 16:36:57 dist-test-slave-65pf krb5kdc[19565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:36:57.730229 19578 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 16:36:57.731220 19578 webserver.cc:460] Webserver started at http://127.0.98.62:46585/ using document root <none> and password file <none>
I20260925 16:36:57.731786 19578 fs_manager.cc:359] Metadata directory not provided
I20260925 16:36:57.731877 19578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:36:57.732107 19578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:36:57.734141 19578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/minicluster-data/master-0/data/instance:
uuid: "12e1a25c6fd64f64818f4baf7e963051"
format_stamp: "Formatted at 2026-09-25 16:36:57 on dist-test-slave-65pf"
server_key: "7b68c8a245c2891cf75f01b7d3717dd0"
server_key_iv: "9c196b99f2f8d65ff9d173fd6f7ed2f7"
server_key_version: "encryptionkey@0"
I20260925 16:36:57.734774 19578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/minicluster-data/master-0/wal/instance:
uuid: "12e1a25c6fd64f64818f4baf7e963051"
format_stamp: "Formatted at 2026-09-25 16:36:57 on dist-test-slave-65pf"
server_key: "7b68c8a245c2891cf75f01b7d3717dd0"
server_key_iv: "9c196b99f2f8d65ff9d173fd6f7ed2f7"
server_key_version: "encryptionkey@0"
I20260925 16:36:57.738409 19578 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 16:36:57.740835 19593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:57.742100 19578 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:36:57.742259 19578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/minicluster-data/master-0/wal
uuid: "12e1a25c6fd64f64818f4baf7e963051"
format_stamp: "Formatted at 2026-09-25 16:36:57 on dist-test-slave-65pf"
server_key: "7b68c8a245c2891cf75f01b7d3717dd0"
server_key_iv: "9c196b99f2f8d65ff9d173fd6f7ed2f7"
server_key_version: "encryptionkey@0"
I20260925 16:36:57.742383 19578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:36:57.756554 19578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:36:57.760046 19578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:36:57.760262 19578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:36:57.767520 19578 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46555
I20260925 16:36:57.767541 19645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46555 every 8 connection(s)
I20260925 16:36:57.768494 19578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:36:57.769878 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 19578
I20260925 16:36:57.770025 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/minicluster-data/master-0/wal/instance
I20260925 16:36:57.770344 392 external_mini_cluster.cc:1428] Setting key 5142e2886fe8a336dd752b9df95b57fa
I20260925 16:36:57.771690 19646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:36:57.777238 19646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12e1a25c6fd64f64818f4baf7e963051: Bootstrap starting.
Sep 25 16:36:57 dist-test-slave-65pf krb5kdc[19565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:36:57.779651 19646 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12e1a25c6fd64f64818f4baf7e963051: Neither blocks nor log segments found. Creating new log.
I20260925 16:36:57.780560 19646 log.cc:824] T 00000000000000000000000000000000 P 12e1a25c6fd64f64818f4baf7e963051: Log is configured to *not* fsync() on all Append() calls
I20260925 16:36:57.782703 19646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12e1a25c6fd64f64818f4baf7e963051: No bootstrap required, opened a new log
I20260925 16:36:57.784766 19649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:57.771841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48998 (local address 127.0.98.62:46555)
0925 16:36:57.772278 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:57.772294 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:57.772329 (+ 35us) server_negotiation.cc:407] Connection header received
0925 16:36:57.772974 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:57.772991 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:57.773322 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:57.773662 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:36:57.774458 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:57.775529 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:57.776418 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:57.776904 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:57.777077 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:57.779582 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:36:57.779588 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:36:57.779600 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:36:57.779627 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:36:57.781664 (+ 2037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:36:57.782168 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:36:57.782188 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:36:57.782193 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:36:57.782313 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:36:57.782687 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:36:57.782693 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:36:57.782697 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:36:57.783197 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:36:57.783420 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:57.783729 (+ 309us) server_negotiation.cc:299] Negotiation successful
0925 16:36:57.784038 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":141,"threads_started":1}
I20260925 16:36:57.786346 19646 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12e1a25c6fd64f64818f4baf7e963051 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12e1a25c6fd64f64818f4baf7e963051" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46555 } }
I20260925 16:36:57.786564 19646 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12e1a25c6fd64f64818f4baf7e963051 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:36:57.786689 19646 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12e1a25c6fd64f64818f4baf7e963051 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e1a25c6fd64f64818f4baf7e963051, State: Initialized, Role: FOLLOWER
I20260925 16:36:57.787135 19646 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12e1a25c6fd64f64818f4baf7e963051 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12e1a25c6fd64f64818f4baf7e963051" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46555 } }
I20260925 16:36:57.787266 19646 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12e1a25c6fd64f64818f4baf7e963051 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:36:57.787307 19646 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12e1a25c6fd64f64818f4baf7e963051 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:36:57.787376 19646 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12e1a25c6fd64f64818f4baf7e963051 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:36:57.788214 19646 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12e1a25c6fd64f64818f4baf7e963051 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12e1a25c6fd64f64818f4baf7e963051" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46555 } }
I20260925 16:36:57.788494 19646 leader_election.cc:302] T 00000000000000000000000000000000 P 12e1a25c6fd64f64818f4baf7e963051 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12e1a25c6fd64f64818f4baf7e963051; no voters:
I20260925 16:36:57.788755 19646 leader_election.cc:288] T 00000000000000000000000000000000 P 12e1a25c6fd64f64818f4baf7e963051 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:36:57.788893 19651 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12e1a25c6fd64f64818f4baf7e963051 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:36:57.789168 19651 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12e1a25c6fd64f64818f4baf7e963051 [term 1 LEADER]: Becoming Leader. State: Replica: 12e1a25c6fd64f64818f4baf7e963051, State: Running, Role: LEADER
I20260925 16:36:57.789526 19651 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12e1a25c6fd64f64818f4baf7e963051 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12e1a25c6fd64f64818f4baf7e963051" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46555 } }
I20260925 16:36:57.789950 19646 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12e1a25c6fd64f64818f4baf7e963051 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:36:57.791163 19652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12e1a25c6fd64f64818f4baf7e963051 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12e1a25c6fd64f64818f4baf7e963051" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12e1a25c6fd64f64818f4baf7e963051" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46555 } } }
I20260925 16:36:57.791313 19652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12e1a25c6fd64f64818f4baf7e963051 [sys.catalog]: This master's current role is: LEADER
I20260925 16:36:57.791302 19653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12e1a25c6fd64f64818f4baf7e963051 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12e1a25c6fd64f64818f4baf7e963051. Latest consensus state: current_term: 1 leader_uuid: "12e1a25c6fd64f64818f4baf7e963051" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12e1a25c6fd64f64818f4baf7e963051" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46555 } } }
I20260925 16:36:57.791652 19653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12e1a25c6fd64f64818f4baf7e963051 [sys.catalog]: This master's current role is: LEADER
I20260925 16:36:57.791852 19660 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:36:57.794509 19660 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:36:57.799872 19660 catalog_manager.cc:1347] Generated new cluster ID: 17a781d7185046a3a3f1ef3ecaff4d08
I20260925 16:36:57.799959 19660 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:36:57.805022 19660 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:36:57.805938 19660 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:36:57.818951 19660 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12e1a25c6fd64f64818f4baf7e963051: Generated new TSK 0
I20260925 16:36:57.819711 19660 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 16:36:57.853611 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46555
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:36:57.966090 19672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:36:57.966374 19672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:36:57.966419 19672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:36:57.970463 19672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:36:57.970534 19672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:36:57.970654 19672 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 16:36:57.975927 19672 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46555
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19672
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:36:57.977030 19672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:36:57.978075 19672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 16:36:57.987931 19672 server_base.cc:1034] running on GCE node
W20260925 16:36:57.987884 19677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:36:57.987884 19678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:36:57.987900 19680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:36:57.988579 19672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:36:57.989156 19672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:36:57.990348 19672 hybrid_clock.cc:648] HybridClock initialized: now 1790354217990330 us; error 28 us; skew 500 ppm
Sep 25 16:36:57 dist-test-slave-65pf krb5kdc[19565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:36:57.993176 19672 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 16:36:57.994046 19672 webserver.cc:460] Webserver started at http://127.0.98.1:37261/ using document root <none> and password file <none>
I20260925 16:36:57.994560 19672 fs_manager.cc:359] Metadata directory not provided
I20260925 16:36:57.994680 19672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:36:57.994853 19672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:36:57.996791 19672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/minicluster-data/ts-0/data/instance:
uuid: "9c5d154e13c04a1dbe56cb0ee26fd4da"
format_stamp: "Formatted at 2026-09-25 16:36:57 on dist-test-slave-65pf"
server_key: "dc22a67e83feba44c6fa8e7ea0842687"
server_key_iv: "10377bffe80c2fe75cc1d5d93c5627c3"
server_key_version: "encryptionkey@0"
I20260925 16:36:57.997292 19672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9c5d154e13c04a1dbe56cb0ee26fd4da"
format_stamp: "Formatted at 2026-09-25 16:36:57 on dist-test-slave-65pf"
server_key: "dc22a67e83feba44c6fa8e7ea0842687"
server_key_iv: "10377bffe80c2fe75cc1d5d93c5627c3"
server_key_version: "encryptionkey@0"
I20260925 16:36:58.000857 19672 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 16:36:58.003189 19687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:58.004392 19672 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:36:58.004487 19672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/minicluster-data/ts-0/wal
uuid: "9c5d154e13c04a1dbe56cb0ee26fd4da"
format_stamp: "Formatted at 2026-09-25 16:36:57 on dist-test-slave-65pf"
server_key: "dc22a67e83feba44c6fa8e7ea0842687"
server_key_iv: "10377bffe80c2fe75cc1d5d93c5627c3"
server_key_version: "encryptionkey@0"
I20260925 16:36:58.004570 19672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:36:58.014834 19672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:36:58.017936 19672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:36:58.018110 19672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:36:58.018733 19672 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:36:58.019570 19672 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:36:58.019621 19672 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:58.019665 19672 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:36:58.019742 19672 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:58.029930 19672 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33617
I20260925 16:36:58.029966 19800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33617 every 8 connection(s)
I20260925 16:36:58.030959 19672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
I20260925 16:36:58.040258 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 19672
Sep 25 16:36:58 dist-test-slave-65pf krb5kdc[19565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:36:58.040424 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/minicluster-data/ts-0/wal/instance
I20260925 16:36:58.040735 392 external_mini_cluster.cc:1428] Setting key f6088c54a9d4906eecd0a4548aae0cad
I20260925 16:36:58.045200 19649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:58.032917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58665 (local address 127.0.98.62:46555)
0925 16:36:58.033078 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:58.033083 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:58.034203 (+ 1120us) server_negotiation.cc:407] Connection header received
0925 16:36:58.035134 (+ 931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:58.035137 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:58.035182 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:58.035261 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:36:58.036424 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:58.037076 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:58.038148 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:58.038378 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:58.038453 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:58.041041 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:36:58.041044 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:36:58.041046 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:36:58.041069 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:36:58.042770 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:36:58.043424 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:36:58.043427 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:36:58.043428 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:36:58.043504 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:36:58.043964 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:36:58.043970 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:36:58.043974 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:36:58.044229 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:36:58.044341 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:58.044872 (+ 531us) server_negotiation.cc:299] Negotiation successful
0925 16:36:58.045027 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 16:36:58.046346 19803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:58.033347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46555 (local address 127.0.98.1:58665)
0925 16:36:58.034022 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:58.034072 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 16:36:58.034944 (+ 872us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:36:58.035429 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:36:58.035436 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:36:58.035791 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:36:58.036277 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:58.036289 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:58.037182 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:58.037184 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:58.038049 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:58.038057 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:58.038503 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:58.038506 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:58.038662 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:58.039300 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:36:58.039316 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:36:58.040871 (+ 1555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:36:58.042886 (+ 2015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:36:58.042896 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:36:58.042916 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:36:58.043307 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:36:58.043598 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:36:58.043604 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:36:58.043609 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:36:58.043834 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:36:58.044341 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:36:58.044346 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:36:58.044612 (+ 266us) client_negotiation.cc:769] Sending connection context
0925 16:36:58.044795 (+ 183us) client_negotiation.cc:240] Negotiation successful
0925 16:36:58.045137 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":114,"threads_started":1}
I20260925 16:36:58.047839 19801 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46555
I20260925 16:36:58.048126 19801 heartbeater.cc:464] Registering TS with master...
I20260925 16:36:58.048815 19801 heartbeater.cc:511] Master 127.0.98.62:46555 requested a full tablet report, sending...
I20260925 16:36:58.050462 19610 ts_manager.cc:194] Registered new tserver with Master: 9c5d154e13c04a1dbe56cb0ee26fd4da (127.0.98.1:33617)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 16:36:58.051734 19610 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58665
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 16:36:58.067723 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46555
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:36:58.177701 19806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:36:58.177974 19806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:36:58.178017 19806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:36:58.182042 19806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:36:58.182106 19806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:36:58.182163 19806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 16:36:58.187403 19806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46555
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:36:58.188458 19806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:36:58.189487 19806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:36:58.199101 19811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:36:58.199133 19806 server_base.cc:1034] running on GCE node
W20260925 16:36:58.199084 19814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:36:58.199084 19812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:36:58.199733 19806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:36:58.200366 19806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:36:58.201530 19806 hybrid_clock.cc:648] HybridClock initialized: now 1790354218201515 us; error 27 us; skew 500 ppm
Sep 25 16:36:58 dist-test-slave-65pf krb5kdc[19565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354218, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:36:58.204363 19806 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 16:36:58.205241 19806 webserver.cc:460] Webserver started at http://127.0.98.2:37739/ using document root <none> and password file <none>
I20260925 16:36:58.205822 19806 fs_manager.cc:359] Metadata directory not provided
I20260925 16:36:58.205883 19806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:36:58.206097 19806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:36:58.208261 19806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/minicluster-data/ts-1/data/instance:
uuid: "41a7d446a4dd42dfa6b7b47e807e28e6"
format_stamp: "Formatted at 2026-09-25 16:36:58 on dist-test-slave-65pf"
server_key: "80fe14b3565b0d2a311c4804566f5bc3"
server_key_iv: "68409a42fcac5b9a5fe570020e627f54"
server_key_version: "encryptionkey@0"
I20260925 16:36:58.208824 19806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/minicluster-data/ts-1/wal/instance:
uuid: "41a7d446a4dd42dfa6b7b47e807e28e6"
format_stamp: "Formatted at 2026-09-25 16:36:58 on dist-test-slave-65pf"
server_key: "80fe14b3565b0d2a311c4804566f5bc3"
server_key_iv: "68409a42fcac5b9a5fe570020e627f54"
server_key_version: "encryptionkey@0"
I20260925 16:36:58.212519 19806 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 16:36:58.214934 19821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:58.216379 19806 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 16:36:58.216497 19806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/minicluster-data/ts-1/wal
uuid: "41a7d446a4dd42dfa6b7b47e807e28e6"
format_stamp: "Formatted at 2026-09-25 16:36:58 on dist-test-slave-65pf"
server_key: "80fe14b3565b0d2a311c4804566f5bc3"
server_key_iv: "68409a42fcac5b9a5fe570020e627f54"
server_key_version: "encryptionkey@0"
I20260925 16:36:58.216631 19806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:36:58.238515 19806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:36:58.241735 19806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:36:58.241896 19806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:36:58.242420 19806 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:36:58.243305 19806 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:36:58.243387 19806 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:58.243459 19806 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:36:58.243512 19806 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:58.252374 19806 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35041
I20260925 16:36:58.252429 19934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35041 every 8 connection(s)
I20260925 16:36:58.253306 19806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
I20260925 16:36:58.254012 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 19806
I20260925 16:36:58.254168 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/minicluster-data/ts-1/wal/instance
I20260925 16:36:58.254472 392 external_mini_cluster.cc:1428] Setting key aad43e997c7127001b36622e7c4571e9
Sep 25 16:36:58 dist-test-slave-65pf krb5kdc[19565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354218, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260925 16:36:58.267346 19649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:58.255309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60119 (local address 127.0.98.62:46555)
0925 16:36:58.255438 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:58.255443 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:58.256383 (+ 940us) server_negotiation.cc:407] Connection header received
0925 16:36:58.257393 (+ 1010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:58.257396 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:58.257449 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:58.257530 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:36:58.258706 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:58.259231 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:58.260496 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:58.260733 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:58.260843 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:58.263324 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:36:58.263327 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:36:58.263330 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:36:58.263355 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:36:58.264956 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:36:58.265554 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:36:58.265560 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:36:58.265565 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:36:58.265659 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:36:58.266064 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:36:58.266067 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:36:58.266069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:36:58.266324 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:36:58.266434 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:58.267024 (+ 590us) server_negotiation.cc:299] Negotiation successful
0925 16:36:58.267157 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 16:36:58.268185 19937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:58.255758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46555 (local address 127.0.98.2:60119)
0925 16:36:58.256238 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:58.256283 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 16:36:58.257187 (+ 904us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:36:58.257701 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:36:58.257707 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:36:58.258060 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:36:58.258537 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:58.258551 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:58.259368 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:58.259373 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:58.260372 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:58.260381 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:58.260880 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:58.260885 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:58.261056 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:58.261661 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:36:58.261677 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:36:58.263142 (+ 1465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:36:58.265079 (+ 1937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:36:58.265086 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:36:58.265123 (+ 37us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:36:58.265396 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:36:58.265763 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:36:58.265766 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:36:58.265768 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:36:58.265913 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:36:58.266431 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:36:58.266438 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:36:58.266774 (+ 336us) client_negotiation.cc:769] Sending connection context
0925 16:36:58.266971 (+ 197us) client_negotiation.cc:240] Negotiation successful
0925 16:36:58.267319 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":154,"threads_started":1}
I20260925 16:36:58.269659 19935 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46555
I20260925 16:36:58.270015 19935 heartbeater.cc:464] Registering TS with master...
I20260925 16:36:58.270566 19935 heartbeater.cc:511] Master 127.0.98.62:46555 requested a full tablet report, sending...
I20260925 16:36:58.271771 19610 ts_manager.cc:194] Registered new tserver with Master: 41a7d446a4dd42dfa6b7b47e807e28e6 (127.0.98.2:35041)
I20260925 16:36:58.272374 19610 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60119
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 16:36:58.282354 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46555
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:36:58.410531 19940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:36:58.410910 19940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:36:58.410990 19940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:36:58.415164 19940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:36:58.415249 19940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:36:58.415336 19940 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 16:36:58.420902 19940 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46555
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19940
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:36:58.422045 19940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:36:58.423158 19940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 16:36:58.432994 19940 server_base.cc:1034] running on GCE node
W20260925 16:36:58.432920 19948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:36:58.432933 19946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:36:58.432914 19945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:36:58.433545 19940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:36:58.434242 19940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:36:58.435448 19940 hybrid_clock.cc:648] HybridClock initialized: now 1790354218435408 us; error 64 us; skew 500 ppm
Sep 25 16:36:58 dist-test-slave-65pf krb5kdc[19565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354218, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:36:58.438238 19940 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 16:36:58.439190 19940 webserver.cc:460] Webserver started at http://127.0.98.3:35099/ using document root <none> and password file <none>
I20260925 16:36:58.439749 19940 fs_manager.cc:359] Metadata directory not provided
I20260925 16:36:58.439853 19940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:36:58.440078 19940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:36:58.442085 19940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/minicluster-data/ts-2/data/instance:
uuid: "c78c4e261ef346e79de56899483a1e78"
format_stamp: "Formatted at 2026-09-25 16:36:58 on dist-test-slave-65pf"
server_key: "df0b1aecbab2097b8fb86a9b92ecce72"
server_key_iv: "93080601af1e38d47a6aa82c23cd00a4"
server_key_version: "encryptionkey@0"
I20260925 16:36:58.442688 19940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c78c4e261ef346e79de56899483a1e78"
format_stamp: "Formatted at 2026-09-25 16:36:58 on dist-test-slave-65pf"
server_key: "df0b1aecbab2097b8fb86a9b92ecce72"
server_key_iv: "93080601af1e38d47a6aa82c23cd00a4"
server_key_version: "encryptionkey@0"
I20260925 16:36:58.446429 19940 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 16:36:58.449131 19955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:58.450402 19940 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:36:58.450532 19940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/minicluster-data/ts-2/wal
uuid: "c78c4e261ef346e79de56899483a1e78"
format_stamp: "Formatted at 2026-09-25 16:36:58 on dist-test-slave-65pf"
server_key: "df0b1aecbab2097b8fb86a9b92ecce72"
server_key_iv: "93080601af1e38d47a6aa82c23cd00a4"
server_key_version: "encryptionkey@0"
I20260925 16:36:58.450712 19940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:36:58.468068 19940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:36:58.471694 19940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:36:58.471926 19940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:36:58.472541 19940 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:36:58.473394 19940 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:36:58.473470 19940 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:58.473538 19940 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:36:58.473572 19940 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:58.482580 19940 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36573
I20260925 16:36:58.482620 20068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36573 every 8 connection(s)
I20260925 16:36:58.483501 19940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
I20260925 16:36:58.489014 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 19940
I20260925 16:36:58.489130 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/minicluster-data/ts-2/wal/instance
I20260925 16:36:58.489421 392 external_mini_cluster.cc:1428] Setting key f52130c690982351a59240b1b8c6e458
Sep 25 16:36:58 dist-test-slave-65pf krb5kdc[19565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354218, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:36:58.496659 19649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:58.485237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46061 (local address 127.0.98.62:46555)
0925 16:36:58.485378 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:58.485382 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:58.486235 (+ 853us) server_negotiation.cc:407] Connection header received
0925 16:36:58.487179 (+ 944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:58.487182 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:58.487253 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:58.487357 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:36:58.488485 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:58.489225 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:58.490406 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:58.490801 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:58.490912 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:58.493511 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:36:58.493515 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:36:58.493518 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:36:58.493542 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:36:58.494937 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:36:58.495380 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:36:58.495382 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:36:58.495383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:36:58.495453 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:36:58.495733 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:36:58.495736 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:36:58.495737 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:36:58.495910 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:36:58.495983 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:58.496445 (+ 462us) server_negotiation.cc:299] Negotiation successful
0925 16:36:58.496540 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 16:36:58.497475 20071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:58.485634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46555 (local address 127.0.98.3:46061)
0925 16:36:58.486067 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:58.486114 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 16:36:58.486977 (+ 863us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:36:58.487522 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:36:58.487528 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:36:58.487887 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:36:58.488345 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:58.488357 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:58.489350 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:58.489352 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:58.490306 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:58.490316 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:58.490957 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:58.490961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:58.491147 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:58.491904 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:36:58.491920 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:36:58.493351 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:36:58.495036 (+ 1685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:36:58.495042 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:36:58.495053 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:36:58.495293 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:36:58.495531 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:36:58.495533 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:36:58.495535 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:36:58.495659 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:36:58.495993 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:36:58.495998 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:36:58.496243 (+ 245us) client_negotiation.cc:769] Sending connection context
0925 16:36:58.496402 (+ 159us) client_negotiation.cc:240] Negotiation successful
0925 16:36:58.496648 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":107,"threads_started":1}
I20260925 16:36:58.498636 20069 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46555
I20260925 16:36:58.498852 20069 heartbeater.cc:464] Registering TS with master...
I20260925 16:36:58.499279 20069 heartbeater.cc:511] Master 127.0.98.62:46555 requested a full tablet report, sending...
I20260925 16:36:58.500253 19610 ts_manager.cc:194] Registered new tserver with Master: c78c4e261ef346e79de56899483a1e78 (127.0.98.3:36573)
I20260925 16:36:58.500833 19610 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46061
I20260925 16:36:58.503412 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 16:36:58.512619 19649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:58.505379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49004 (local address 127.0.98.62:46555)
0925 16:36:58.505542 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:58.505547 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:58.505705 (+ 158us) server_negotiation.cc:407] Connection header received
0925 16:36:58.505853 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:58.505858 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:58.505900 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:58.505982 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:36:58.506689 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:58.507205 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:58.508090 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:58.508308 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:58.508415 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:58.509738 (+ 1323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:36:58.509741 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:36:58.509744 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:36:58.509774 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:36:58.510985 (+ 1211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:36:58.511394 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:36:58.511399 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:36:58.511401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:36:58.511478 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:36:58.511750 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:36:58.511755 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:36:58.511757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:36:58.511975 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:36:58.512118 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:58.512347 (+ 229us) server_negotiation.cc:299] Negotiation successful
0925 16:36:58.512455 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 16:36:58.515456 19610 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49004:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 16:36:58.518431 19610 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:36:58.539129 20083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:58.533339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35041 (local address 127.0.0.1:53404)
0925 16:36:58.534080 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:58.534095 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:36:58.534185 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:36:58.535234 (+ 1049us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:36:58.535237 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:36:58.535252 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:36:58.535364 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:58.535371 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:58.536397 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:58.536400 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:58.537571 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:58.537579 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:58.538523 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:58.538527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:58.538668 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:58.538713 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 16:36:58.538832 (+ 119us) client_negotiation.cc:240] Negotiation successful
0925 16:36:58.538968 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"mutex_wait_us":1,"thread_start_us":111,"threads_started":1}
I20260925 16:36:58.539152 20085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:58.534419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53404 (local address 127.0.98.2:35041)
0925 16:36:58.534785 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:58.534798 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:58.534821 (+ 23us) server_negotiation.cc:407] Connection header received
0925 16:36:58.534900 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:58.534905 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:58.535096 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:58.535242 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:36:58.535441 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:58.536261 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:58.537697 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:58.538398 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:58.538504 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:58.538679 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:58.538877 (+ 198us) server_negotiation.cc:299] Negotiation successful
0925 16:36:58.538983 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":113,"threads_started":1}
I20260925 16:36:58.539727 20081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:58.533185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36573 (local address 127.0.0.1:39012)
0925 16:36:58.533692 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:58.533741 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 16:36:58.533896 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:36:58.535053 (+ 1157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:36:58.535059 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:36:58.535082 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:36:58.535238 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:58.535244 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:58.536709 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:58.536712 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:58.537948 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:58.537958 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:58.539175 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:58.539181 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:58.539332 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:58.539421 (+ 89us) client_negotiation.cc:769] Sending connection context
0925 16:36:58.539492 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 16:36:58.539592 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"mutex_wait_us":40,"thread_start_us":79,"threads_started":1}
I20260925 16:36:58.540162 20082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:58.533333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39012 (local address 127.0.98.3:36573)
0925 16:36:58.534613 (+ 1280us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:58.534626 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:58.534649 (+ 23us) server_negotiation.cc:407] Connection header received
0925 16:36:58.534733 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:58.534737 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:58.534900 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:58.535064 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:36:58.535365 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:58.536562 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:58.538073 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:58.538885 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:58.538982 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:58.539108 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:58.539909 (+ 801us) server_negotiation.cc:299] Negotiation successful
0925 16:36:58.540006 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1149,"thread_start_us":125,"threads_started":1}
I20260925 16:36:58.541106 20080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:58.533787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33617 (local address 127.0.0.1:43560)
0925 16:36:58.534231 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:58.534244 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:36:58.534311 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:36:58.535407 (+ 1096us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:36:58.535411 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:36:58.535430 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:36:58.535535 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:58.535541 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:58.536881 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:58.536884 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:58.537995 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:58.538005 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:58.539155 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:58.539161 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:58.539265 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:58.539434 (+ 169us) client_negotiation.cc:769] Sending connection context
0925 16:36:58.539505 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 16:36:58.541015 (+ 1510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"mutex_wait_us":1531,"thread_start_us":102,"threads_started":1}
I20260925 16:36:58.541514 20084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:58.534081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43560 (local address 127.0.98.1:33617)
0925 16:36:58.534785 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:58.534799 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:58.534833 (+ 34us) server_negotiation.cc:407] Connection header received
0925 16:36:58.534900 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:58.534905 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:58.535096 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:58.535242 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:36:58.535663 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:58.536748 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:58.538111 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:58.538881 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:58.538972 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:58.539094 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:58.539598 (+ 504us) server_negotiation.cc:299] Negotiation successful
0925 16:36:58.541360 (+ 1762us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":73,"threads_started":1}
I20260925 16:36:58.542842 20003 tablet_service.cc:1467] Processing CreateTablet for tablet bdec96c77549495bae2675fe2f1f5c76 (DEFAULT_TABLE table=test-table [id=876c5b1c136246bb9c4c29962e80d226]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:36:58.542835 19869 tablet_service.cc:1467] Processing CreateTablet for tablet bdec96c77549495bae2675fe2f1f5c76 (DEFAULT_TABLE table=test-table [id=876c5b1c136246bb9c4c29962e80d226]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:36:58.543998 19869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdec96c77549495bae2675fe2f1f5c76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:36:58.543998 20003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdec96c77549495bae2675fe2f1f5c76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:36:58.543790 19735 tablet_service.cc:1467] Processing CreateTablet for tablet bdec96c77549495bae2675fe2f1f5c76 (DEFAULT_TABLE table=test-table [id=876c5b1c136246bb9c4c29962e80d226]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:36:58.544744 19735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdec96c77549495bae2675fe2f1f5c76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:36:58.551082 20086 tablet_bootstrap.cc:492] T bdec96c77549495bae2675fe2f1f5c76 P 9c5d154e13c04a1dbe56cb0ee26fd4da: Bootstrap starting.
I20260925 16:36:58.552317 20087 tablet_bootstrap.cc:492] T bdec96c77549495bae2675fe2f1f5c76 P c78c4e261ef346e79de56899483a1e78: Bootstrap starting.
I20260925 16:36:58.552328 20088 tablet_bootstrap.cc:492] T bdec96c77549495bae2675fe2f1f5c76 P 41a7d446a4dd42dfa6b7b47e807e28e6: Bootstrap starting.
I20260925 16:36:58.553212 20086 tablet_bootstrap.cc:654] T bdec96c77549495bae2675fe2f1f5c76 P 9c5d154e13c04a1dbe56cb0ee26fd4da: Neither blocks nor log segments found. Creating new log.
I20260925 16:36:58.553961 20086 log.cc:824] T bdec96c77549495bae2675fe2f1f5c76 P 9c5d154e13c04a1dbe56cb0ee26fd4da: Log is configured to *not* fsync() on all Append() calls
I20260925 16:36:58.554430 20087 tablet_bootstrap.cc:654] T bdec96c77549495bae2675fe2f1f5c76 P c78c4e261ef346e79de56899483a1e78: Neither blocks nor log segments found. Creating new log.
I20260925 16:36:58.554491 20088 tablet_bootstrap.cc:654] T bdec96c77549495bae2675fe2f1f5c76 P 41a7d446a4dd42dfa6b7b47e807e28e6: Neither blocks nor log segments found. Creating new log.
I20260925 16:36:58.555708 20087 log.cc:824] T bdec96c77549495bae2675fe2f1f5c76 P c78c4e261ef346e79de56899483a1e78: Log is configured to *not* fsync() on all Append() calls
I20260925 16:36:58.555708 20088 log.cc:824] T bdec96c77549495bae2675fe2f1f5c76 P 41a7d446a4dd42dfa6b7b47e807e28e6: Log is configured to *not* fsync() on all Append() calls
I20260925 16:36:58.556162 20086 tablet_bootstrap.cc:492] T bdec96c77549495bae2675fe2f1f5c76 P 9c5d154e13c04a1dbe56cb0ee26fd4da: No bootstrap required, opened a new log
I20260925 16:36:58.556355 20086 ts_tablet_manager.cc:1397] T bdec96c77549495bae2675fe2f1f5c76 P 9c5d154e13c04a1dbe56cb0ee26fd4da: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 16:36:58.557549 20087 tablet_bootstrap.cc:492] T bdec96c77549495bae2675fe2f1f5c76 P c78c4e261ef346e79de56899483a1e78: No bootstrap required, opened a new log
I20260925 16:36:58.557675 20087 ts_tablet_manager.cc:1397] T bdec96c77549495bae2675fe2f1f5c76 P c78c4e261ef346e79de56899483a1e78: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 16:36:58.557958 20088 tablet_bootstrap.cc:492] T bdec96c77549495bae2675fe2f1f5c76 P 41a7d446a4dd42dfa6b7b47e807e28e6: No bootstrap required, opened a new log
I20260925 16:36:58.558146 20088 ts_tablet_manager.cc:1397] T bdec96c77549495bae2675fe2f1f5c76 P 41a7d446a4dd42dfa6b7b47e807e28e6: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260925 16:36:58.560499 20087 raft_consensus.cc:348] T bdec96c77549495bae2675fe2f1f5c76 P c78c4e261ef346e79de56899483a1e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c78c4e261ef346e79de56899483a1e78" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36573 } } peers { permanent_uuid: "41a7d446a4dd42dfa6b7b47e807e28e6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35041 } } peers { permanent_uuid: "9c5d154e13c04a1dbe56cb0ee26fd4da" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33617 } }
I20260925 16:36:58.560745 20087 raft_consensus.cc:374] T bdec96c77549495bae2675fe2f1f5c76 P c78c4e261ef346e79de56899483a1e78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:36:58.560794 20087 raft_consensus.cc:729] T bdec96c77549495bae2675fe2f1f5c76 P c78c4e261ef346e79de56899483a1e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c78c4e261ef346e79de56899483a1e78, State: Initialized, Role: FOLLOWER
I20260925 16:36:58.560788 20086 raft_consensus.cc:348] T bdec96c77549495bae2675fe2f1f5c76 P 9c5d154e13c04a1dbe56cb0ee26fd4da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c78c4e261ef346e79de56899483a1e78" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36573 } } peers { permanent_uuid: "41a7d446a4dd42dfa6b7b47e807e28e6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35041 } } peers { permanent_uuid: "9c5d154e13c04a1dbe56cb0ee26fd4da" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33617 } }
I20260925 16:36:58.561033 20086 raft_consensus.cc:374] T bdec96c77549495bae2675fe2f1f5c76 P 9c5d154e13c04a1dbe56cb0ee26fd4da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:36:58.561089 20086 raft_consensus.cc:729] T bdec96c77549495bae2675fe2f1f5c76 P 9c5d154e13c04a1dbe56cb0ee26fd4da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c5d154e13c04a1dbe56cb0ee26fd4da, State: Initialized, Role: FOLLOWER
I20260925 16:36:58.561255 20087 consensus_queue.cc:260] T bdec96c77549495bae2675fe2f1f5c76 P c78c4e261ef346e79de56899483a1e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c78c4e261ef346e79de56899483a1e78" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36573 } } peers { permanent_uuid: "41a7d446a4dd42dfa6b7b47e807e28e6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35041 } } peers { permanent_uuid: "9c5d154e13c04a1dbe56cb0ee26fd4da" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33617 } }
I20260925 16:36:58.561492 20086 consensus_queue.cc:260] T bdec96c77549495bae2675fe2f1f5c76 P 9c5d154e13c04a1dbe56cb0ee26fd4da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c78c4e261ef346e79de56899483a1e78" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36573 } } peers { permanent_uuid: "41a7d446a4dd42dfa6b7b47e807e28e6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35041 } } peers { permanent_uuid: "9c5d154e13c04a1dbe56cb0ee26fd4da" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33617 } }
I20260925 16:36:58.561971 20069 heartbeater.cc:503] Master 127.0.98.62:46555 was elected leader, sending a full tablet report...
I20260925 16:36:58.561856 20088 raft_consensus.cc:348] T bdec96c77549495bae2675fe2f1f5c76 P 41a7d446a4dd42dfa6b7b47e807e28e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c78c4e261ef346e79de56899483a1e78" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36573 } } peers { permanent_uuid: "41a7d446a4dd42dfa6b7b47e807e28e6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35041 } } peers { permanent_uuid: "9c5d154e13c04a1dbe56cb0ee26fd4da" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33617 } }
I20260925 16:36:58.562078 20088 raft_consensus.cc:374] T bdec96c77549495bae2675fe2f1f5c76 P 41a7d446a4dd42dfa6b7b47e807e28e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:36:58.562129 20088 raft_consensus.cc:729] T bdec96c77549495bae2675fe2f1f5c76 P 41a7d446a4dd42dfa6b7b47e807e28e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41a7d446a4dd42dfa6b7b47e807e28e6, State: Initialized, Role: FOLLOWER
I20260925 16:36:58.562287 20087 ts_tablet_manager.cc:1428] T bdec96c77549495bae2675fe2f1f5c76 P c78c4e261ef346e79de56899483a1e78: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 16:36:58.562389 20086 ts_tablet_manager.cc:1428] T bdec96c77549495bae2675fe2f1f5c76 P 9c5d154e13c04a1dbe56cb0ee26fd4da: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 16:36:58.562708 19801 heartbeater.cc:503] Master 127.0.98.62:46555 was elected leader, sending a full tablet report...
I20260925 16:36:58.563279 20088 consensus_queue.cc:260] T bdec96c77549495bae2675fe2f1f5c76 P 41a7d446a4dd42dfa6b7b47e807e28e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c78c4e261ef346e79de56899483a1e78" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36573 } } peers { permanent_uuid: "41a7d446a4dd42dfa6b7b47e807e28e6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35041 } } peers { permanent_uuid: "9c5d154e13c04a1dbe56cb0ee26fd4da" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33617 } }
I20260925 16:36:58.563910 19935 heartbeater.cc:503] Master 127.0.98.62:46555 was elected leader, sending a full tablet report...
I20260925 16:36:58.564200 20088 ts_tablet_manager.cc:1428] T bdec96c77549495bae2675fe2f1f5c76 P 41a7d446a4dd42dfa6b7b47e807e28e6: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260925 16:36:58.691694 20092 raft_consensus.cc:482] T bdec96c77549495bae2675fe2f1f5c76 P c78c4e261ef346e79de56899483a1e78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:36:58.691951 20092 raft_consensus.cc:504] T bdec96c77549495bae2675fe2f1f5c76 P c78c4e261ef346e79de56899483a1e78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c78c4e261ef346e79de56899483a1e78" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36573 } } peers { permanent_uuid: "41a7d446a4dd42dfa6b7b47e807e28e6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35041 } } peers { permanent_uuid: "9c5d154e13c04a1dbe56cb0ee26fd4da" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33617 } }
I20260925 16:36:58.693142 20092 leader_election.cc:288] T bdec96c77549495bae2675fe2f1f5c76 P c78c4e261ef346e79de56899483a1e78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41a7d446a4dd42dfa6b7b47e807e28e6 (127.0.98.2:35041), 9c5d154e13c04a1dbe56cb0ee26fd4da (127.0.98.1:33617)
I20260925 16:36:58.697008 20085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:58.693510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51651 (local address 127.0.98.2:35041)
0925 16:36:58.693695 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:58.693700 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:58.693732 (+ 32us) server_negotiation.cc:407] Connection header received
0925 16:36:58.693793 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:58.693796 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:58.693857 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:58.693940 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:36:58.694237 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:58.694890 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:58.695967 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:58.696437 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:58.696524 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:58.696612 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:58.696802 (+ 190us) server_negotiation.cc:299] Negotiation successful
0925 16:36:58.696876 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260925 16:36:58.697021 20084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:58.693522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52143 (local address 127.0.98.1:33617)
0925 16:36:58.693668 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:58.693672 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:58.693903 (+ 231us) server_negotiation.cc:407] Connection header received
0925 16:36:58.694008 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:58.694011 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:58.694055 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:58.694137 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:36:58.694445 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:58.695036 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:58.696070 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:58.696465 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:58.696588 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:58.696649 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:58.696856 (+ 207us) server_negotiation.cc:299] Negotiation successful
0925 16:36:58.696921 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 16:36:58.697010 20071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:58.693388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35041 (local address 127.0.98.3:51651)
0925 16:36:58.693579 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:58.693597 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 16:36:58.693692 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:36:58.693969 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:36:58.693972 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:36:58.693990 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:36:58.694123 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:58.694132 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:58.694998 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:58.695001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:58.695838 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:58.695847 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:58.696556 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:58.696561 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:58.696663 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:58.696690 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 16:36:58.696759 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 16:36:58.696867 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":1}
I20260925 16:36:58.697017 20095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:58.693418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33617 (local address 127.0.98.3:52143)
0925 16:36:58.693805 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:58.693818 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:36:58.693919 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:36:58.694158 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:36:58.694161 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:36:58.694193 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:36:58.694321 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:58.694327 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:58.695143 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:58.695145 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:58.695951 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:58.695961 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:58.696610 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:58.696613 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:58.696709 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:58.696727 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 16:36:58.696821 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 16:36:58.696884 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":129,"threads_started":1}
I20260925 16:36:58.697757 19755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdec96c77549495bae2675fe2f1f5c76" candidate_uuid: "c78c4e261ef346e79de56899483a1e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c5d154e13c04a1dbe56cb0ee26fd4da" is_pre_election: true
I20260925 16:36:58.697757 19889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdec96c77549495bae2675fe2f1f5c76" candidate_uuid: "c78c4e261ef346e79de56899483a1e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41a7d446a4dd42dfa6b7b47e807e28e6" is_pre_election: true
I20260925 16:36:58.698072 19755 raft_consensus.cc:2454] T bdec96c77549495bae2675fe2f1f5c76 P 9c5d154e13c04a1dbe56cb0ee26fd4da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c78c4e261ef346e79de56899483a1e78 in term 0.
I20260925 16:36:58.698072 19889 raft_consensus.cc:2454] T bdec96c77549495bae2675fe2f1f5c76 P 41a7d446a4dd42dfa6b7b47e807e28e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c78c4e261ef346e79de56899483a1e78 in term 0.
I20260925 16:36:58.698598 19959 leader_election.cc:302] T bdec96c77549495bae2675fe2f1f5c76 P c78c4e261ef346e79de56899483a1e78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 41a7d446a4dd42dfa6b7b47e807e28e6, 9c5d154e13c04a1dbe56cb0ee26fd4da, c78c4e261ef346e79de56899483a1e78; no voters:
I20260925 16:36:58.698913 20092 raft_consensus.cc:2781] T bdec96c77549495bae2675fe2f1f5c76 P c78c4e261ef346e79de56899483a1e78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:36:58.699005 20092 raft_consensus.cc:482] T bdec96c77549495bae2675fe2f1f5c76 P c78c4e261ef346e79de56899483a1e78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:36:58.699054 20092 raft_consensus.cc:3037] T bdec96c77549495bae2675fe2f1f5c76 P c78c4e261ef346e79de56899483a1e78 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:36:58.700040 20092 raft_consensus.cc:504] T bdec96c77549495bae2675fe2f1f5c76 P c78c4e261ef346e79de56899483a1e78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c78c4e261ef346e79de56899483a1e78" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36573 } } peers { permanent_uuid: "41a7d446a4dd42dfa6b7b47e807e28e6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35041 } } peers { permanent_uuid: "9c5d154e13c04a1dbe56cb0ee26fd4da" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33617 } }
I20260925 16:36:58.700383 20092 leader_election.cc:288] T bdec96c77549495bae2675fe2f1f5c76 P c78c4e261ef346e79de56899483a1e78 [CANDIDATE]: Term 1 election: Requested vote from peers 41a7d446a4dd42dfa6b7b47e807e28e6 (127.0.98.2:35041), 9c5d154e13c04a1dbe56cb0ee26fd4da (127.0.98.1:33617)
I20260925 16:36:58.700752 19755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdec96c77549495bae2675fe2f1f5c76" candidate_uuid: "c78c4e261ef346e79de56899483a1e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c5d154e13c04a1dbe56cb0ee26fd4da"
I20260925 16:36:58.700752 19889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdec96c77549495bae2675fe2f1f5c76" candidate_uuid: "c78c4e261ef346e79de56899483a1e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41a7d446a4dd42dfa6b7b47e807e28e6"
I20260925 16:36:58.700889 19889 raft_consensus.cc:3037] T bdec96c77549495bae2675fe2f1f5c76 P 41a7d446a4dd42dfa6b7b47e807e28e6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:36:58.700878 19755 raft_consensus.cc:3037] T bdec96c77549495bae2675fe2f1f5c76 P 9c5d154e13c04a1dbe56cb0ee26fd4da [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:36:58.702165 19889 raft_consensus.cc:2454] T bdec96c77549495bae2675fe2f1f5c76 P 41a7d446a4dd42dfa6b7b47e807e28e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c78c4e261ef346e79de56899483a1e78 in term 1.
I20260925 16:36:58.702165 19755 raft_consensus.cc:2454] T bdec96c77549495bae2675fe2f1f5c76 P 9c5d154e13c04a1dbe56cb0ee26fd4da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c78c4e261ef346e79de56899483a1e78 in term 1.
I20260925 16:36:58.702461 19959 leader_election.cc:302] T bdec96c77549495bae2675fe2f1f5c76 P c78c4e261ef346e79de56899483a1e78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c5d154e13c04a1dbe56cb0ee26fd4da, c78c4e261ef346e79de56899483a1e78; no voters:
I20260925 16:36:58.702649 20092 raft_consensus.cc:2781] T bdec96c77549495bae2675fe2f1f5c76 P c78c4e261ef346e79de56899483a1e78 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:36:58.702853 20092 raft_consensus.cc:686] T bdec96c77549495bae2675fe2f1f5c76 P c78c4e261ef346e79de56899483a1e78 [term 1 LEADER]: Becoming Leader. State: Replica: c78c4e261ef346e79de56899483a1e78, State: Running, Role: LEADER
I20260925 16:36:58.703226 20092 consensus_queue.cc:237] T bdec96c77549495bae2675fe2f1f5c76 P c78c4e261ef346e79de56899483a1e78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c78c4e261ef346e79de56899483a1e78" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36573 } } peers { permanent_uuid: "41a7d446a4dd42dfa6b7b47e807e28e6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35041 } } peers { permanent_uuid: "9c5d154e13c04a1dbe56cb0ee26fd4da" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33617 } }
I20260925 16:36:58.706454 19610 catalog_manager.cc:5495] T bdec96c77549495bae2675fe2f1f5c76 P c78c4e261ef346e79de56899483a1e78 reported cstate change: term changed from 0 to 1, leader changed from <none> to c78c4e261ef346e79de56899483a1e78 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "c78c4e261ef346e79de56899483a1e78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c78c4e261ef346e79de56899483a1e78" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36573 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "41a7d446a4dd42dfa6b7b47e807e28e6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c5d154e13c04a1dbe56cb0ee26fd4da" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33617 } health_report { overall_health: UNKNOWN } } }
I20260925 16:36:58.728727 20082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:58.725322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39028 (local address 127.0.98.3:36573)
0925 16:36:58.725458 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:58.725461 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:58.725474 (+ 13us) server_negotiation.cc:407] Connection header received
0925 16:36:58.725589 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:58.725591 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:58.725645 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:58.725744 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:36:58.726013 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:58.726517 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:58.727385 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:58.727606 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:58.727685 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:58.727918 (+ 233us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:36:58.728334 (+ 416us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:36:58.728423 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:58.728472 (+ 49us) server_negotiation.cc:299] Negotiation successful
0925 16:36:58.728520 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260925 16:36:58.734366 20070 tablet.cc:2369] T bdec96c77549495bae2675fe2f1f5c76 P c78c4e261ef346e79de56899483a1e78: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:36:58.734407 19889 raft_consensus.cc:1261] T bdec96c77549495bae2675fe2f1f5c76 P 41a7d446a4dd42dfa6b7b47e807e28e6 [term 1 FOLLOWER]: Refusing update from remote peer c78c4e261ef346e79de56899483a1e78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:36:58.734520 19755 raft_consensus.cc:1261] T bdec96c77549495bae2675fe2f1f5c76 P 9c5d154e13c04a1dbe56cb0ee26fd4da [term 1 FOLLOWER]: Refusing update from remote peer c78c4e261ef346e79de56899483a1e78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:36:58.735191 20096 consensus_queue.cc:1035] T bdec96c77549495bae2675fe2f1f5c76 P c78c4e261ef346e79de56899483a1e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c5d154e13c04a1dbe56cb0ee26fd4da" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:36:58.735353 20092 consensus_queue.cc:1035] T bdec96c77549495bae2675fe2f1f5c76 P c78c4e261ef346e79de56899483a1e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41a7d446a4dd42dfa6b7b47e807e28e6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:36:58.742388 20101 mvcc.cc:205] Tried to move back new op lower bound from 7333290879929077760 to 7333290879810977792. Current Snapshot: MvccSnapshot[applied={T|T < 7333290879929077760}]
I20260925 16:36:58.745026 20100 mvcc.cc:205] Tried to move back new op lower bound from 7333290879929077760 to 7333290879810977792. Current Snapshot: MvccSnapshot[applied={T|T < 7333290879929077760}]
I20260925 16:36:58.750326 19608 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:36:58.750500 19608 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:36:58.753026 19608 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 12e1a25c6fd64f64818f4baf7e963051: Sending DeleteTablet for 3 replicas of tablet bdec96c77549495bae2675fe2f1f5c76
I20260925 16:36:58.753772 20003 tablet_service.cc:1514] Processing DeleteTablet for tablet bdec96c77549495bae2675fe2f1f5c76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:36:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39012
I20260925 16:36:58.753804 19869 tablet_service.cc:1514] Processing DeleteTablet for tablet bdec96c77549495bae2675fe2f1f5c76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:36:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53404
I20260925 16:36:58.753830 19735 tablet_service.cc:1514] Processing DeleteTablet for tablet bdec96c77549495bae2675fe2f1f5c76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:36:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43560
I20260925 16:36:58.754338 20109 tablet_replica.cc:330] T bdec96c77549495bae2675fe2f1f5c76 P c78c4e261ef346e79de56899483a1e78: stopping tablet replica
I20260925 16:36:58.754508 20111 tablet_replica.cc:330] T bdec96c77549495bae2675fe2f1f5c76 P 41a7d446a4dd42dfa6b7b47e807e28e6: stopping tablet replica
I20260925 16:36:58.754722 20109 raft_consensus.cc:2229] T bdec96c77549495bae2675fe2f1f5c76 P c78c4e261ef346e79de56899483a1e78 [term 1 LEADER]: Raft consensus shutting down.
I20260925 16:36:58.754781 20110 tablet_replica.cc:330] T bdec96c77549495bae2675fe2f1f5c76 P 9c5d154e13c04a1dbe56cb0ee26fd4da: stopping tablet replica
I20260925 16:36:58.755050 20110 raft_consensus.cc:2229] T bdec96c77549495bae2675fe2f1f5c76 P 9c5d154e13c04a1dbe56cb0ee26fd4da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:36:58.755294 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 19672
I20260925 16:36:58.756222 20111 raft_consensus.cc:2229] T bdec96c77549495bae2675fe2f1f5c76 P 41a7d446a4dd42dfa6b7b47e807e28e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:36:58.756517 20111 raft_consensus.cc:2258] T bdec96c77549495bae2675fe2f1f5c76 P 41a7d446a4dd42dfa6b7b47e807e28e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:36:58.759280 20109 raft_consensus.cc:2258] T bdec96c77549495bae2675fe2f1f5c76 P c78c4e261ef346e79de56899483a1e78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:36:58.760849 20109 ts_tablet_manager.cc:1905] T bdec96c77549495bae2675fe2f1f5c76 P c78c4e261ef346e79de56899483a1e78: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 16:36:58.761817 19597 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33617 (error 108)
I20260925 16:36:58.762909 20080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:58.762557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33617 (local address 127.0.0.1:43566)
0925 16:36:58.762725 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:58.762804 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260925 16:36:58.762964 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 19806
I20260925 16:36:58.764936 20109 ts_tablet_manager.cc:1918] T bdec96c77549495bae2675fe2f1f5c76 P c78c4e261ef346e79de56899483a1e78: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:36:58.765045 20109 log.cc:1196] T bdec96c77549495bae2675fe2f1f5c76 P c78c4e261ef346e79de56899483a1e78: Deleting WAL directory at /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353966814071-392-0/minicluster-data/ts-2/wal/wals/bdec96c77549495bae2675fe2f1f5c76
I20260925 16:36:58.765405 20109 ts_tablet_manager.cc:1939] T bdec96c77549495bae2675fe2f1f5c76 P c78c4e261ef346e79de56899483a1e78: Deleting consensus metadata
I20260925 16:36:58.766752 19594 catalog_manager.cc:4841] TS c78c4e261ef346e79de56899483a1e78 (127.0.98.3:36573): tablet bdec96c77549495bae2675fe2f1f5c76 (table test-table [id=876c5b1c136246bb9c4c29962e80d226]) successfully deleted
W20260925 16:36:58.770293 19597 catalog_manager.cc:4584] TS 9c5d154e13c04a1dbe56cb0ee26fd4da (127.0.98.1:33617): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bdec96c77549495bae2675fe2f1f5c76: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33617: connect: Connection refused (error 111)
I20260925 16:36:58.770790 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 19940
I20260925 16:36:58.771886 20080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:58.770966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35041 (local address 127.0.0.1:53406)
0925 16:36:58.771761 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:58.771826 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:35041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":719}
I20260925 16:36:58.776857 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 19578
2026-09-25T16:36:58Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2334 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-25T16:36:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:36:58Z Disabled control of system clock
I20260925 16:36:58.806910 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353966814071-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353966814071-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353966814071-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35227
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35227
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 16:36:58.917943 20116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:36:58.918212 20116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:36:58.918249 20116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:36:58.922245 20116 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:36:58.922305 20116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:36:58.922320 20116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:36:58.922338 20116 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:36:58.922353 20116 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:36:58.927136 20116 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353966814071-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353966814071-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35227
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353966814071-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:36:58.928184 20116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:36:58.929509 20116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:36:58.938097 20121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:36:58.938129 20122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:36:58.938135 20124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:36:58.938304 20116 server_base.cc:1034] running on GCE node
I20260925 16:36:58.939064 20116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:36:58.940167 20116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:36:58.941358 20116 hybrid_clock.cc:648] HybridClock initialized: now 1790354218941328 us; error 47 us; skew 500 ppm
I20260925 16:36:58.943208 20116 webserver.cc:460] Webserver started at http://127.0.98.62:39171/ using document root <none> and password file <none>
I20260925 16:36:58.943791 20116 fs_manager.cc:359] Metadata directory not provided
I20260925 16:36:58.943884 20116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:36:58.944114 20116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:36:58.946151 20116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353966814071-392-0/minicluster-data/master-0/data/instance:
uuid: "5d8bedda64854c23846f54d85ce8bbdf"
format_stamp: "Formatted at 2026-09-25 16:36:58 on dist-test-slave-65pf"
server_key: "d871dee9bcbf6f4bb5748a3a5b0c1e56"
server_key_iv: "439de1689ae8888d58be3067892d5f75"
server_key_version: "encryptionkey@0"
I20260925 16:36:58.946770 20116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353966814071-392-0/minicluster-data/master-0/wal/instance:
uuid: "5d8bedda64854c23846f54d85ce8bbdf"
format_stamp: "Formatted at 2026-09-25 16:36:58 on dist-test-slave-65pf"
server_key: "d871dee9bcbf6f4bb5748a3a5b0c1e56"
server_key_iv: "439de1689ae8888d58be3067892d5f75"
server_key_version: "encryptionkey@0"
I20260925 16:36:58.950346 20116 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 16:36:58.952898 20130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:58.954151 20116 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:36:58.954295 20116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353966814071-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353966814071-392-0/minicluster-data/master-0/wal
uuid: "5d8bedda64854c23846f54d85ce8bbdf"
format_stamp: "Formatted at 2026-09-25 16:36:58 on dist-test-slave-65pf"
server_key: "d871dee9bcbf6f4bb5748a3a5b0c1e56"
server_key_iv: "439de1689ae8888d58be3067892d5f75"
server_key_version: "encryptionkey@0"
I20260925 16:36:58.954418 20116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353966814071-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353966814071-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353966814071-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:36:58.963613 20116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:36:58.964254 20116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:36:58.964450 20116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:36:58.971881 20116 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35227
I20260925 16:36:58.971894 20182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35227 every 8 connection(s)
I20260925 16:36:58.972829 20116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:36:58.975863 20183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:36:58.981266 20183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d8bedda64854c23846f54d85ce8bbdf: Bootstrap starting.
I20260925 16:36:58.982776 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 20116
I20260925 16:36:58.982913 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353966814071-392-0/minicluster-data/master-0/wal/instance
I20260925 16:36:58.983201 392 external_mini_cluster.cc:1428] Setting key f25bf4c3969545619f5ea0107126347c
I20260925 16:36:58.983971 20183 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d8bedda64854c23846f54d85ce8bbdf: Neither blocks nor log segments found. Creating new log.
I20260925 16:36:58.984879 20183 log.cc:824] T 00000000000000000000000000000000 P 5d8bedda64854c23846f54d85ce8bbdf: Log is configured to *not* fsync() on all Append() calls
I20260925 16:36:58.987089 20183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d8bedda64854c23846f54d85ce8bbdf: No bootstrap required, opened a new log
I20260925 16:36:58.991307 20183 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d8bedda64854c23846f54d85ce8bbdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d8bedda64854c23846f54d85ce8bbdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35227 } }
I20260925 16:36:58.991595 20183 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d8bedda64854c23846f54d85ce8bbdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:36:58.991664 20183 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d8bedda64854c23846f54d85ce8bbdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d8bedda64854c23846f54d85ce8bbdf, State: Initialized, Role: FOLLOWER
I20260925 16:36:58.992085 20183 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d8bedda64854c23846f54d85ce8bbdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d8bedda64854c23846f54d85ce8bbdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35227 } }
I20260925 16:36:58.992276 20183 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d8bedda64854c23846f54d85ce8bbdf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:36:58.992357 20183 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d8bedda64854c23846f54d85ce8bbdf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:36:58.992479 20183 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d8bedda64854c23846f54d85ce8bbdf [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:36:58.992748 20186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:58.984416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35394 (local address 127.0.98.62:35227)
0925 16:36:58.985060 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:58.985082 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:58.985122 (+ 40us) server_negotiation.cc:407] Connection header received
0925 16:36:58.985957 (+ 835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:58.985978 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:58.986308 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:58.986652 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:36:58.986998 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:58.988007 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:58.989235 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:58.989693 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:58.989807 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:58.990741 (+ 934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:36:58.990746 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:36:58.990759 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:36:58.991147 (+ 388us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:36:58.991205 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:36:58.991224 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:36:58.991299 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:58.991473 (+ 174us) server_negotiation.cc:299] Negotiation successful
0925 16:36:58.991770 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":127,"threads_started":1}
I20260925 16:36:58.993640 20183 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d8bedda64854c23846f54d85ce8bbdf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d8bedda64854c23846f54d85ce8bbdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35227 } }
I20260925 16:36:58.994000 20183 leader_election.cc:302] T 00000000000000000000000000000000 P 5d8bedda64854c23846f54d85ce8bbdf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d8bedda64854c23846f54d85ce8bbdf; no voters:
I20260925 16:36:58.994307 20183 leader_election.cc:288] T 00000000000000000000000000000000 P 5d8bedda64854c23846f54d85ce8bbdf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:36:58.994422 20188 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d8bedda64854c23846f54d85ce8bbdf [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:36:58.994649 20188 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d8bedda64854c23846f54d85ce8bbdf [term 1 LEADER]: Becoming Leader. State: Replica: 5d8bedda64854c23846f54d85ce8bbdf, State: Running, Role: LEADER
I20260925 16:36:58.994952 20188 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d8bedda64854c23846f54d85ce8bbdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d8bedda64854c23846f54d85ce8bbdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35227 } }
I20260925 16:36:58.995566 20183 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d8bedda64854c23846f54d85ce8bbdf [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:36:58.996069 20188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d8bedda64854c23846f54d85ce8bbdf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d8bedda64854c23846f54d85ce8bbdf. Latest consensus state: current_term: 1 leader_uuid: "5d8bedda64854c23846f54d85ce8bbdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d8bedda64854c23846f54d85ce8bbdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35227 } } }
I20260925 16:36:58.996188 20188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d8bedda64854c23846f54d85ce8bbdf [sys.catalog]: This master's current role is: LEADER
I20260925 16:36:58.996452 20189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d8bedda64854c23846f54d85ce8bbdf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d8bedda64854c23846f54d85ce8bbdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d8bedda64854c23846f54d85ce8bbdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35227 } } }
I20260925 16:36:58.996544 20189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d8bedda64854c23846f54d85ce8bbdf [sys.catalog]: This master's current role is: LEADER
I20260925 16:36:58.996604 20193 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:36:58.999737 20193 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:36:59.005275 20193 catalog_manager.cc:1347] Generated new cluster ID: 1b133a6e873f4dcf8af6442a5b87b91b
I20260925 16:36:59.005358 20193 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:36:59.013113 20193 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:36:59.014015 20193 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:36:59.031204 20193 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d8bedda64854c23846f54d85ce8bbdf: Generated new TSK 0
I20260925 16:36:59.031956 20193 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:36:59.041040 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353966814071-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353966814071-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353966814071-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35227
--builtin_ntp_servers=127.0.98.20:34353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 16:36:59.151301 20207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:36:59.151552 20207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:36:59.151590 20207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:36:59.155567 20207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:36:59.155622 20207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:36:59.155678 20207 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 16:36:59.161080 20207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353966814071-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353966814071-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20207
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353966814071-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:36:59.162156 20207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:36:59.163576 20207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:36:59.173018 20212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:36:59.173038 20213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:36:59.173077 20215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:36:59.173318 20207 server_base.cc:1034] running on GCE node
I20260925 16:36:59.173831 20207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:36:59.174432 20207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:36:59.175612 20207 hybrid_clock.cc:648] HybridClock initialized: now 1790354219175584 us; error 50 us; skew 500 ppm
I20260925 16:36:59.177428 20207 webserver.cc:460] Webserver started at http://127.0.98.1:45845/ using document root <none> and password file <none>
I20260925 16:36:59.178020 20207 fs_manager.cc:359] Metadata directory not provided
I20260925 16:36:59.178107 20207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:36:59.178346 20207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:36:59.180399 20207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353966814071-392-0/minicluster-data/ts-0/data/instance:
uuid: "cbcf3fa3797145778c907290027e807b"
format_stamp: "Formatted at 2026-09-25 16:36:59 on dist-test-slave-65pf"
server_key: "6ac787a021e6202ec9193b75bc0e021f"
server_key_iv: "9d86030b91acfa45c1541f5a29d1a17b"
server_key_version: "encryptionkey@0"
I20260925 16:36:59.180946 20207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353966814071-392-0/minicluster-data/ts-0/wal/instance:
uuid: "cbcf3fa3797145778c907290027e807b"
format_stamp: "Formatted at 2026-09-25 16:36:59 on dist-test-slave-65pf"
server_key: "6ac787a021e6202ec9193b75bc0e021f"
server_key_iv: "9d86030b91acfa45c1541f5a29d1a17b"
server_key_version: "encryptionkey@0"
I20260925 16:36:59.184617 20207 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 16:36:59.187198 20221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:59.188472 20207 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:36:59.188612 20207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353966814071-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353966814071-392-0/minicluster-data/ts-0/wal
uuid: "cbcf3fa3797145778c907290027e807b"
format_stamp: "Formatted at 2026-09-25 16:36:59 on dist-test-slave-65pf"
server_key: "6ac787a021e6202ec9193b75bc0e021f"
server_key_iv: "9d86030b91acfa45c1541f5a29d1a17b"
server_key_version: "encryptionkey@0"
I20260925 16:36:59.188740 20207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353966814071-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353966814071-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353966814071-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:36:59.201658 20207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:36:59.202332 20207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:36:59.202526 20207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:36:59.203128 20207 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:36:59.204028 20207 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:36:59.204097 20207 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:59.204165 20207 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:36:59.204217 20207 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:59.213261 20207 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38255
I20260925 16:36:59.213290 20334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38255 every 8 connection(s)
I20260925 16:36:59.214151 20207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353966814071-392-0/minicluster-data/ts-0/data/info.pb
I20260925 16:36:59.217350 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 20207
I20260925 16:36:59.217483 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353966814071-392-0/minicluster-data/ts-0/wal/instance
I20260925 16:36:59.217796 392 external_mini_cluster.cc:1428] Setting key 40edad8a0bcc0a04e333115f96242835
I20260925 16:36:59.219764 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353966814071-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353966814071-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353966814071-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35227
--builtin_ntp_servers=127.0.98.20:34353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 16:36:59.223724 20186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:59.216058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49961 (local address 127.0.98.62:35227)
0925 16:36:59.216205 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:59.216210 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:59.217351 (+ 1141us) server_negotiation.cc:407] Connection header received
0925 16:36:59.218943 (+ 1592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:59.218946 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:59.218999 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:59.219079 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:36:59.220038 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:59.220578 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:59.221718 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:59.221953 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:59.222063 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:59.223220 (+ 1157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:36:59.223224 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:36:59.223227 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:36:59.223242 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:36:59.223262 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:36:59.223277 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:36:59.223367 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:59.223526 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 16:36:59.223585 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 16:36:59.224610 20337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:59.216516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35227 (local address 127.0.98.1:49961)
0925 16:36:59.217179 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:59.217220 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 16:36:59.218719 (+ 1499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:36:59.219244 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:36:59.219251 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:36:59.219337 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:36:59.219809 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:59.219852 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:59.220711 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:59.220715 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:59.221588 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:59.221596 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:59.222116 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:59.222121 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:59.222288 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:59.222959 (+ 671us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 16:36:59.222961 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:36:59.223000 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 16:36:59.223064 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:36:59.223368 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:36:59.223379 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:36:59.223387 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 16:36:59.223525 (+ 138us) client_negotiation.cc:240] Negotiation successful
0925 16:36:59.223886 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":102,"threads_started":1}
I20260925 16:36:59.225863 20335 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35227
I20260925 16:36:59.226114 20335 heartbeater.cc:464] Registering TS with master...
I20260925 16:36:59.226678 20335 heartbeater.cc:511] Master 127.0.98.62:35227 requested a full tablet report, sending...
I20260925 16:36:59.228346 20147 ts_manager.cc:194] Registered new tserver with Master: cbcf3fa3797145778c907290027e807b (127.0.98.1:38255)
I20260925 16:36:59.230085 20147 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:49961
W20260925 16:36:59.330384 20338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:36:59.330679 20338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:36:59.330718 20338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:36:59.334733 20338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:36:59.334818 20338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:36:59.334913 20338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 16:36:59.339637 20338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353966814071-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353966814071-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353966814071-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:36:59.340646 20338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:36:59.341941 20338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:36:59.351538 20346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:36:59.351552 20344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:36:59.351544 20343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:36:59.351874 20338 server_base.cc:1034] running on GCE node
I20260925 16:36:59.352278 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:36:59.352919 20338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:36:59.354115 20338 hybrid_clock.cc:648] HybridClock initialized: now 1790354219354076 us; error 53 us; skew 500 ppm
I20260925 16:36:59.356005 20338 webserver.cc:460] Webserver started at http://127.0.98.2:45643/ using document root <none> and password file <none>
I20260925 16:36:59.356587 20338 fs_manager.cc:359] Metadata directory not provided
I20260925 16:36:59.356680 20338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:36:59.356912 20338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:36:59.359144 20338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353966814071-392-0/minicluster-data/ts-1/data/instance:
uuid: "330aa80ec1ea4f0785a33dfe4b1c916f"
format_stamp: "Formatted at 2026-09-25 16:36:59 on dist-test-slave-65pf"
server_key: "a8f6da8d47acdb509313b67c02d5f815"
server_key_iv: "a2968f1755e1b73555f7663a32ee8eb7"
server_key_version: "encryptionkey@0"
I20260925 16:36:59.359714 20338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353966814071-392-0/minicluster-data/ts-1/wal/instance:
uuid: "330aa80ec1ea4f0785a33dfe4b1c916f"
format_stamp: "Formatted at 2026-09-25 16:36:59 on dist-test-slave-65pf"
server_key: "a8f6da8d47acdb509313b67c02d5f815"
server_key_iv: "a2968f1755e1b73555f7663a32ee8eb7"
server_key_version: "encryptionkey@0"
I20260925 16:36:59.363353 20338 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 16:36:59.365696 20352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:59.366918 20338 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:36:59.367043 20338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353966814071-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353966814071-392-0/minicluster-data/ts-1/wal
uuid: "330aa80ec1ea4f0785a33dfe4b1c916f"
format_stamp: "Formatted at 2026-09-25 16:36:59 on dist-test-slave-65pf"
server_key: "a8f6da8d47acdb509313b67c02d5f815"
server_key_iv: "a2968f1755e1b73555f7663a32ee8eb7"
server_key_version: "encryptionkey@0"
I20260925 16:36:59.367151 20338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353966814071-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353966814071-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353966814071-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:36:59.389938 20338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:36:59.390640 20338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:36:59.390827 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:36:59.391484 20338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:36:59.392403 20338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:36:59.392446 20338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:59.392484 20338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:36:59.392499 20338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:59.402033 20338 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36627
I20260925 16:36:59.402091 20465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36627 every 8 connection(s)
I20260925 16:36:59.403052 20338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353966814071-392-0/minicluster-data/ts-1/data/info.pb
I20260925 16:36:59.406198 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 20338
I20260925 16:36:59.406339 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353966814071-392-0/minicluster-data/ts-1/wal/instance
I20260925 16:36:59.406669 392 external_mini_cluster.cc:1428] Setting key 82dcf0a76d86f17ab9399c5628ffd23f
I20260925 16:36:59.408767 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353966814071-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353966814071-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353966814071-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35227
--builtin_ntp_servers=127.0.98.20:34353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 16:36:59.412953 20186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:59.404844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38401 (local address 127.0.98.62:35227)
0925 16:36:59.404999 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:59.405003 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:59.406081 (+ 1078us) server_negotiation.cc:407] Connection header received
0925 16:36:59.407835 (+ 1754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:59.407838 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:59.407881 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:59.407974 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:36:59.408907 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:59.409431 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:59.411087 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:59.411304 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:59.411389 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:59.412482 (+ 1093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:36:59.412491 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:36:59.412494 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:36:59.412505 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:36:59.412522 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:36:59.412530 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:36:59.412603 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:59.412762 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 16:36:59.412819 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 16:36:59.413765 20468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:59.405257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35227 (local address 127.0.98.2:38401)
0925 16:36:59.405895 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:59.405964 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0925 16:36:59.407616 (+ 1652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:36:59.408172 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:36:59.408181 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:36:59.408295 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:36:59.408767 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:59.408779 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:59.409559 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:59.409567 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:59.410961 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:59.410973 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:59.411432 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:59.411435 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:59.411604 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:59.412239 (+ 635us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 16:36:59.412240 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:36:59.412276 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 16:36:59.412337 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:36:59.412619 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:36:59.412629 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:36:59.412641 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 16:36:59.412762 (+ 121us) client_negotiation.cc:240] Negotiation successful
0925 16:36:59.413046 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":115,"threads_started":1}
I20260925 16:36:59.414908 20466 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35227
I20260925 16:36:59.415149 20466 heartbeater.cc:464] Registering TS with master...
I20260925 16:36:59.415642 20466 heartbeater.cc:511] Master 127.0.98.62:35227 requested a full tablet report, sending...
I20260925 16:36:59.416669 20147 ts_manager.cc:194] Registered new tserver with Master: 330aa80ec1ea4f0785a33dfe4b1c916f (127.0.98.2:36627)
I20260925 16:36:59.417516 20147 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:38401
W20260925 16:36:59.520617 20469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:36:59.520885 20469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:36:59.520923 20469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:36:59.524938 20469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:36:59.524991 20469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:36:59.525045 20469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 16:36:59.529654 20469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353966814071-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353966814071-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353966814071-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:36:59.530779 20469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:36:59.532142 20469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:36:59.541708 20475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:36:59.541814 20474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:36:59.541721 20477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:36:59.541884 20469 server_base.cc:1034] running on GCE node
I20260925 16:36:59.542445 20469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:36:59.543083 20469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:36:59.544261 20469 hybrid_clock.cc:648] HybridClock initialized: now 1790354219544215 us; error 60 us; skew 500 ppm
I20260925 16:36:59.545997 20469 webserver.cc:460] Webserver started at http://127.0.98.3:43067/ using document root <none> and password file <none>
I20260925 16:36:59.546572 20469 fs_manager.cc:359] Metadata directory not provided
I20260925 16:36:59.546733 20469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:36:59.546955 20469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:36:59.548952 20469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353966814071-392-0/minicluster-data/ts-2/data/instance:
uuid: "caf932fb540e4ddd8a78ad228da6c97a"
format_stamp: "Formatted at 2026-09-25 16:36:59 on dist-test-slave-65pf"
server_key: "986d138b75f1c4d01603bebde58589c4"
server_key_iv: "afbb3ce038134185489b61ea73b893b0"
server_key_version: "encryptionkey@0"
I20260925 16:36:59.549504 20469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353966814071-392-0/minicluster-data/ts-2/wal/instance:
uuid: "caf932fb540e4ddd8a78ad228da6c97a"
format_stamp: "Formatted at 2026-09-25 16:36:59 on dist-test-slave-65pf"
server_key: "986d138b75f1c4d01603bebde58589c4"
server_key_iv: "afbb3ce038134185489b61ea73b893b0"
server_key_version: "encryptionkey@0"
I20260925 16:36:59.553053 20469 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 16:36:59.555459 20483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:59.556650 20469 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:36:59.556777 20469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353966814071-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353966814071-392-0/minicluster-data/ts-2/wal
uuid: "caf932fb540e4ddd8a78ad228da6c97a"
format_stamp: "Formatted at 2026-09-25 16:36:59 on dist-test-slave-65pf"
server_key: "986d138b75f1c4d01603bebde58589c4"
server_key_iv: "afbb3ce038134185489b61ea73b893b0"
server_key_version: "encryptionkey@0"
I20260925 16:36:59.556886 20469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353966814071-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353966814071-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353966814071-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:36:59.573518 20469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:36:59.574215 20469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:36:59.574419 20469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:36:59.575064 20469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:36:59.576001 20469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:36:59.576079 20469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:59.576149 20469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:36:59.576184 20469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:36:59.584815 20469 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46047
I20260925 16:36:59.584846 20596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46047 every 8 connection(s)
I20260925 16:36:59.585741 20469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353966814071-392-0/minicluster-data/ts-2/data/info.pb
I20260925 16:36:59.595043 20186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:59.587660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49273 (local address 127.0.98.62:35227)
0925 16:36:59.587794 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:59.587798 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:59.588801 (+ 1003us) server_negotiation.cc:407] Connection header received
0925 16:36:59.590248 (+ 1447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:59.590252 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:59.590310 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:59.590417 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:36:59.591339 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:59.591873 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:59.593238 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:59.593450 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:59.593531 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:59.594659 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:36:59.594662 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:36:59.594664 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:36:59.594676 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:36:59.594691 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:36:59.594698 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:36:59.594751 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:59.594882 (+ 131us) server_negotiation.cc:299] Negotiation successful
0925 16:36:59.594929 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 16:36:59.595878 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 20469
I20260925 16:36:59.595994 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353966814071-392-0/minicluster-data/ts-2/wal/instance
I20260925 16:36:59.596024 20599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:59.588012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35227 (local address 127.0.98.3:49273)
0925 16:36:59.588654 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:59.588693 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 16:36:59.590006 (+ 1313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:36:59.590630 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:36:59.590644 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:36:59.590727 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:36:59.591196 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:59.591209 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:59.592067 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:59.592070 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:59.592990 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:59.593000 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:59.593610 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:59.593613 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:59.593751 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:59.594393 (+ 642us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 16:36:59.594395 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:36:59.594434 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 16:36:59.594494 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:36:59.594765 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:36:59.594772 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:36:59.594776 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 16:36:59.594868 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 16:36:59.595076 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":101,"threads_started":1}
I20260925 16:36:59.596328 392 external_mini_cluster.cc:1428] Setting key b24739a15fdbeefa3c299497cfafa3ee
I20260925 16:36:59.597365 20597 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35227
I20260925 16:36:59.597668 20597 heartbeater.cc:464] Registering TS with master...
I20260925 16:36:59.598232 20597 heartbeater.cc:511] Master 127.0.98.62:35227 requested a full tablet report, sending...
I20260925 16:36:59.599267 20147 ts_manager.cc:194] Registered new tserver with Master: caf932fb540e4ddd8a78ad228da6c97a (127.0.98.3:46047)
I20260925 16:36:59.599826 20147 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:49273
I20260925 16:36:59.610308 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 16:36:59.616144 20186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:59.612416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35406 (local address 127.0.98.62:35227)
0925 16:36:59.612575 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:59.612579 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:59.612727 (+ 148us) server_negotiation.cc:407] Connection header received
0925 16:36:59.612829 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:59.612831 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:59.612872 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:59.612962 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:36:59.613210 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:59.613771 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:59.614652 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:59.614928 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:59.615028 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:59.615776 (+ 748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:36:59.615779 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:36:59.615781 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:36:59.615794 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:36:59.615813 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:36:59.615821 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:36:59.615884 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:59.615961 (+ 77us) server_negotiation.cc:299] Negotiation successful
0925 16:36:59.616009 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 16:36:59.619222 20147 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35406:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 16:36:59.622346 20147 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:36:59.636983 20610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:59.631362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46047 (local address 127.0.0.1:55450)
0925 16:36:59.632611 (+ 1249us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:59.632623 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 16:36:59.632721 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:36:59.633217 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:36:59.633220 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:36:59.633236 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:36:59.633325 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:59.633342 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:59.634299 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:59.634304 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:59.635669 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:59.635678 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:59.636442 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:59.636445 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:59.636540 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:59.636573 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 16:36:59.636685 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 16:36:59.636819 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1191,"mutex_wait_us":99,"thread_start_us":96,"threads_started":1}
I20260925 16:36:59.636991 20613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:59.631713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55450 (local address 127.0.98.3:46047)
0925 16:36:59.632466 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:59.632480 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:59.632817 (+ 337us) server_negotiation.cc:407] Connection header received
0925 16:36:59.632900 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:59.632905 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:59.633089 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:59.633228 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:36:59.633449 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:59.634173 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:59.635793 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:59.636321 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:59.636440 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:59.636635 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:59.636714 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 16:36:59.636823 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653,"thread_start_us":167,"threads_started":1}
I20260925 16:36:59.637429 20611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:59.630927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36627 (local address 127.0.0.1:34646)
0925 16:36:59.631748 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:59.631799 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 16:36:59.631938 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:36:59.632571 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:36:59.632577 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:36:59.632602 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:36:59.632780 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:59.632789 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:59.634487 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:59.634490 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:59.635680 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:59.635691 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:59.637025 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:59.637029 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:59.637118 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:59.637141 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 16:36:59.637220 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 16:36:59.637288 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"mutex_wait_us":72,"thread_start_us":90,"threads_started":1}
I20260925 16:36:59.637651 20609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:59.631051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34646 (local address 127.0.98.2:36627)
0925 16:36:59.631451 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:59.631462 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:59.632082 (+ 620us) server_negotiation.cc:407] Connection header received
0925 16:36:59.632167 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:59.632172 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:59.632406 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:59.632564 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:36:59.632922 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:59.634327 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:59.635809 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:59.636650 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:59.636767 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:59.636958 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:59.637278 (+ 320us) server_negotiation.cc:299] Negotiation successful
0925 16:36:59.637388 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":103,"threads_started":1}
I20260925 16:36:59.638435 20608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:59.631074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38255 (local address 127.0.0.1:58194)
0925 16:36:59.632105 (+ 1031us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:59.632120 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:36:59.632226 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:36:59.632786 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:36:59.632792 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:36:59.632809 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:36:59.632903 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:59.632908 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:59.634449 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:59.634452 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:59.636912 (+ 2460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:59.636923 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:59.638045 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:59.638048 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:59.638156 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:59.638176 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 16:36:59.638250 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 16:36:59.638306 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":962,"mutex_wait_us":1200,"thread_start_us":111,"threads_started":1}
I20260925 16:36:59.639091 20612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:59.631763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58194 (local address 127.0.98.1:38255)
0925 16:36:59.632181 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:59.632196 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:59.632220 (+ 24us) server_negotiation.cc:407] Connection header received
0925 16:36:59.632394 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:59.632400 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:59.632623 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:59.632798 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:36:59.633044 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:59.634327 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:59.637060 (+ 2733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:59.637920 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:59.638014 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:59.638151 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:59.638307 (+ 156us) server_negotiation.cc:299] Negotiation successful
0925 16:36:59.638897 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":104,"threads_started":1}
I20260925 16:36:59.640861 20531 tablet_service.cc:1467] Processing CreateTablet for tablet a3f8d6ac7d5c4cc2b1b795c908d81b60 (DEFAULT_TABLE table=test-table [id=88903894c57d4821acfc0dbb2b87f958]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:36:59.640851 20400 tablet_service.cc:1467] Processing CreateTablet for tablet a3f8d6ac7d5c4cc2b1b795c908d81b60 (DEFAULT_TABLE table=test-table [id=88903894c57d4821acfc0dbb2b87f958]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:36:59.641423 20269 tablet_service.cc:1467] Processing CreateTablet for tablet a3f8d6ac7d5c4cc2b1b795c908d81b60 (DEFAULT_TABLE table=test-table [id=88903894c57d4821acfc0dbb2b87f958]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:36:59.641959 20400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3f8d6ac7d5c4cc2b1b795c908d81b60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:36:59.641959 20531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3f8d6ac7d5c4cc2b1b795c908d81b60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:36:59.642499 20269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3f8d6ac7d5c4cc2b1b795c908d81b60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:36:59.649003 20614 tablet_bootstrap.cc:492] T a3f8d6ac7d5c4cc2b1b795c908d81b60 P cbcf3fa3797145778c907290027e807b: Bootstrap starting.
I20260925 16:36:59.650223 20616 tablet_bootstrap.cc:492] T a3f8d6ac7d5c4cc2b1b795c908d81b60 P 330aa80ec1ea4f0785a33dfe4b1c916f: Bootstrap starting.
I20260925 16:36:59.650883 20615 tablet_bootstrap.cc:492] T a3f8d6ac7d5c4cc2b1b795c908d81b60 P caf932fb540e4ddd8a78ad228da6c97a: Bootstrap starting.
I20260925 16:36:59.651602 20614 tablet_bootstrap.cc:654] T a3f8d6ac7d5c4cc2b1b795c908d81b60 P cbcf3fa3797145778c907290027e807b: Neither blocks nor log segments found. Creating new log.
I20260925 16:36:59.652390 20616 tablet_bootstrap.cc:654] T a3f8d6ac7d5c4cc2b1b795c908d81b60 P 330aa80ec1ea4f0785a33dfe4b1c916f: Neither blocks nor log segments found. Creating new log.
I20260925 16:36:59.652489 20615 tablet_bootstrap.cc:654] T a3f8d6ac7d5c4cc2b1b795c908d81b60 P caf932fb540e4ddd8a78ad228da6c97a: Neither blocks nor log segments found. Creating new log.
I20260925 16:36:59.652516 20614 log.cc:824] T a3f8d6ac7d5c4cc2b1b795c908d81b60 P cbcf3fa3797145778c907290027e807b: Log is configured to *not* fsync() on all Append() calls
I20260925 16:36:59.653228 20616 log.cc:824] T a3f8d6ac7d5c4cc2b1b795c908d81b60 P 330aa80ec1ea4f0785a33dfe4b1c916f: Log is configured to *not* fsync() on all Append() calls
I20260925 16:36:59.653286 20615 log.cc:824] T a3f8d6ac7d5c4cc2b1b795c908d81b60 P caf932fb540e4ddd8a78ad228da6c97a: Log is configured to *not* fsync() on all Append() calls
I20260925 16:36:59.655287 20615 tablet_bootstrap.cc:492] T a3f8d6ac7d5c4cc2b1b795c908d81b60 P caf932fb540e4ddd8a78ad228da6c97a: No bootstrap required, opened a new log
I20260925 16:36:59.655494 20615 ts_tablet_manager.cc:1397] T a3f8d6ac7d5c4cc2b1b795c908d81b60 P caf932fb540e4ddd8a78ad228da6c97a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 16:36:59.655505 20616 tablet_bootstrap.cc:492] T a3f8d6ac7d5c4cc2b1b795c908d81b60 P 330aa80ec1ea4f0785a33dfe4b1c916f: No bootstrap required, opened a new log
I20260925 16:36:59.655737 20616 ts_tablet_manager.cc:1397] T a3f8d6ac7d5c4cc2b1b795c908d81b60 P 330aa80ec1ea4f0785a33dfe4b1c916f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 16:36:59.655985 20614 tablet_bootstrap.cc:492] T a3f8d6ac7d5c4cc2b1b795c908d81b60 P cbcf3fa3797145778c907290027e807b: No bootstrap required, opened a new log
I20260925 16:36:59.656209 20614 ts_tablet_manager.cc:1397] T a3f8d6ac7d5c4cc2b1b795c908d81b60 P cbcf3fa3797145778c907290027e807b: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 16:36:59.659107 20615 raft_consensus.cc:348] T a3f8d6ac7d5c4cc2b1b795c908d81b60 P caf932fb540e4ddd8a78ad228da6c97a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "330aa80ec1ea4f0785a33dfe4b1c916f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36627 } } peers { permanent_uuid: "cbcf3fa3797145778c907290027e807b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38255 } } peers { permanent_uuid: "caf932fb540e4ddd8a78ad228da6c97a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46047 } }
I20260925 16:36:59.659339 20615 raft_consensus.cc:374] T a3f8d6ac7d5c4cc2b1b795c908d81b60 P caf932fb540e4ddd8a78ad228da6c97a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:36:59.659392 20615 raft_consensus.cc:729] T a3f8d6ac7d5c4cc2b1b795c908d81b60 P caf932fb540e4ddd8a78ad228da6c97a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: caf932fb540e4ddd8a78ad228da6c97a, State: Initialized, Role: FOLLOWER
I20260925 16:36:59.660013 20615 consensus_queue.cc:260] T a3f8d6ac7d5c4cc2b1b795c908d81b60 P caf932fb540e4ddd8a78ad228da6c97a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "330aa80ec1ea4f0785a33dfe4b1c916f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36627 } } peers { permanent_uuid: "cbcf3fa3797145778c907290027e807b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38255 } } peers { permanent_uuid: "caf932fb540e4ddd8a78ad228da6c97a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46047 } }
I20260925 16:36:59.660331 20614 raft_consensus.cc:348] T a3f8d6ac7d5c4cc2b1b795c908d81b60 P cbcf3fa3797145778c907290027e807b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "330aa80ec1ea4f0785a33dfe4b1c916f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36627 } } peers { permanent_uuid: "cbcf3fa3797145778c907290027e807b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38255 } } peers { permanent_uuid: "caf932fb540e4ddd8a78ad228da6c97a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46047 } }
I20260925 16:36:59.660652 20614 raft_consensus.cc:374] T a3f8d6ac7d5c4cc2b1b795c908d81b60 P cbcf3fa3797145778c907290027e807b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:36:59.660728 20614 raft_consensus.cc:729] T a3f8d6ac7d5c4cc2b1b795c908d81b60 P cbcf3fa3797145778c907290027e807b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbcf3fa3797145778c907290027e807b, State: Initialized, Role: FOLLOWER
I20260925 16:36:59.661082 20615 ts_tablet_manager.cc:1428] T a3f8d6ac7d5c4cc2b1b795c908d81b60 P caf932fb540e4ddd8a78ad228da6c97a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 16:36:59.660934 20616 raft_consensus.cc:348] T a3f8d6ac7d5c4cc2b1b795c908d81b60 P 330aa80ec1ea4f0785a33dfe4b1c916f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "330aa80ec1ea4f0785a33dfe4b1c916f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36627 } } peers { permanent_uuid: "cbcf3fa3797145778c907290027e807b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38255 } } peers { permanent_uuid: "caf932fb540e4ddd8a78ad228da6c97a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46047 } }
I20260925 16:36:59.661161 20616 raft_consensus.cc:374] T a3f8d6ac7d5c4cc2b1b795c908d81b60 P 330aa80ec1ea4f0785a33dfe4b1c916f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:36:59.661218 20616 raft_consensus.cc:729] T a3f8d6ac7d5c4cc2b1b795c908d81b60 P 330aa80ec1ea4f0785a33dfe4b1c916f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 330aa80ec1ea4f0785a33dfe4b1c916f, State: Initialized, Role: FOLLOWER
I20260925 16:36:59.661159 20614 consensus_queue.cc:260] T a3f8d6ac7d5c4cc2b1b795c908d81b60 P cbcf3fa3797145778c907290027e807b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "330aa80ec1ea4f0785a33dfe4b1c916f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36627 } } peers { permanent_uuid: "cbcf3fa3797145778c907290027e807b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38255 } } peers { permanent_uuid: "caf932fb540e4ddd8a78ad228da6c97a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46047 } }
I20260925 16:36:59.661453 20597 heartbeater.cc:503] Master 127.0.98.62:35227 was elected leader, sending a full tablet report...
I20260925 16:36:59.661623 20616 consensus_queue.cc:260] T a3f8d6ac7d5c4cc2b1b795c908d81b60 P 330aa80ec1ea4f0785a33dfe4b1c916f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "330aa80ec1ea4f0785a33dfe4b1c916f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36627 } } peers { permanent_uuid: "cbcf3fa3797145778c907290027e807b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38255 } } peers { permanent_uuid: "caf932fb540e4ddd8a78ad228da6c97a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46047 } }
I20260925 16:36:59.662220 20614 ts_tablet_manager.cc:1428] T a3f8d6ac7d5c4cc2b1b795c908d81b60 P cbcf3fa3797145778c907290027e807b: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 16:36:59.662242 20466 heartbeater.cc:503] Master 127.0.98.62:35227 was elected leader, sending a full tablet report...
I20260925 16:36:59.662519 20616 ts_tablet_manager.cc:1428] T a3f8d6ac7d5c4cc2b1b795c908d81b60 P 330aa80ec1ea4f0785a33dfe4b1c916f: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260925 16:36:59.662533 20335 heartbeater.cc:503] Master 127.0.98.62:35227 was elected leader, sending a full tablet report...
W20260925 16:36:59.715372 20336 tablet.cc:2369] T a3f8d6ac7d5c4cc2b1b795c908d81b60 P cbcf3fa3797145778c907290027e807b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:36:59.734712 20620 raft_consensus.cc:482] T a3f8d6ac7d5c4cc2b1b795c908d81b60 P caf932fb540e4ddd8a78ad228da6c97a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:36:59.734922 20620 raft_consensus.cc:504] T a3f8d6ac7d5c4cc2b1b795c908d81b60 P caf932fb540e4ddd8a78ad228da6c97a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "330aa80ec1ea4f0785a33dfe4b1c916f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36627 } } peers { permanent_uuid: "cbcf3fa3797145778c907290027e807b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38255 } } peers { permanent_uuid: "caf932fb540e4ddd8a78ad228da6c97a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46047 } }
I20260925 16:36:59.736024 20620 leader_election.cc:288] T a3f8d6ac7d5c4cc2b1b795c908d81b60 P caf932fb540e4ddd8a78ad228da6c97a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 330aa80ec1ea4f0785a33dfe4b1c916f (127.0.98.2:36627), cbcf3fa3797145778c907290027e807b (127.0.98.1:38255)
I20260925 16:36:59.740247 20609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:59.736317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60423 (local address 127.0.98.2:36627)
0925 16:36:59.736427 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:59.736434 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:59.736550 (+ 116us) server_negotiation.cc:407] Connection header received
0925 16:36:59.736621 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:59.736623 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:59.736712 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:59.736797 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:36:59.737214 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:59.738033 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:59.739285 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:59.739665 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:59.739742 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:59.739819 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:59.740042 (+ 223us) server_negotiation.cc:299] Negotiation successful
0925 16:36:59.740113 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 16:36:59.740260 20623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:59.736295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38255 (local address 127.0.98.3:49499)
0925 16:36:59.736621 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:59.736641 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 16:36:59.736712 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:36:59.736942 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:36:59.736945 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:36:59.736959 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:36:59.737108 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:59.737114 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:59.738163 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:59.738167 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:59.739153 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:59.739163 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:59.739858 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:59.739866 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:59.739938 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:59.739955 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 16:36:59.740051 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 16:36:59.740112 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":21,"thread_start_us":104,"threads_started":1}
I20260925 16:36:59.740275 20612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:59.736403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49499 (local address 127.0.98.1:38255)
0925 16:36:59.736525 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:59.736532 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:59.736713 (+ 181us) server_negotiation.cc:407] Connection header received
0925 16:36:59.736772 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:59.736777 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:59.736844 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:59.736922 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:36:59.737252 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:59.738004 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:59.739279 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:59.739695 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:59.739835 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:59.739902 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:59.740082 (+ 180us) server_negotiation.cc:299] Negotiation successful
0925 16:36:59.740167 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 16:36:59.740266 20599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:59.736191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36627 (local address 127.0.98.3:60423)
0925 16:36:59.736407 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:59.736422 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:36:59.736530 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:36:59.736843 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:36:59.736846 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:36:59.736858 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:36:59.737025 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:59.737033 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:59.738239 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:59.738243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:59.739148 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:36:59.739158 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:36:59.739783 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:36:59.739788 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:36:59.739890 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:59.739919 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 16:36:59.739998 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 16:36:59.740096 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":1}
I20260925 16:36:59.741024 20289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3f8d6ac7d5c4cc2b1b795c908d81b60" candidate_uuid: "caf932fb540e4ddd8a78ad228da6c97a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbcf3fa3797145778c907290027e807b" is_pre_election: true
I20260925 16:36:59.741045 20420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3f8d6ac7d5c4cc2b1b795c908d81b60" candidate_uuid: "caf932fb540e4ddd8a78ad228da6c97a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "330aa80ec1ea4f0785a33dfe4b1c916f" is_pre_election: true
I20260925 16:36:59.741317 20289 raft_consensus.cc:2454] T a3f8d6ac7d5c4cc2b1b795c908d81b60 P cbcf3fa3797145778c907290027e807b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate caf932fb540e4ddd8a78ad228da6c97a in term 0.
I20260925 16:36:59.741321 20420 raft_consensus.cc:2454] T a3f8d6ac7d5c4cc2b1b795c908d81b60 P 330aa80ec1ea4f0785a33dfe4b1c916f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate caf932fb540e4ddd8a78ad228da6c97a in term 0.
I20260925 16:36:59.741842 20484 leader_election.cc:302] T a3f8d6ac7d5c4cc2b1b795c908d81b60 P caf932fb540e4ddd8a78ad228da6c97a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 330aa80ec1ea4f0785a33dfe4b1c916f, caf932fb540e4ddd8a78ad228da6c97a, cbcf3fa3797145778c907290027e807b; no voters:
I20260925 16:36:59.742118 20620 raft_consensus.cc:2781] T a3f8d6ac7d5c4cc2b1b795c908d81b60 P caf932fb540e4ddd8a78ad228da6c97a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:36:59.742211 20620 raft_consensus.cc:482] T a3f8d6ac7d5c4cc2b1b795c908d81b60 P caf932fb540e4ddd8a78ad228da6c97a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:36:59.742266 20620 raft_consensus.cc:3037] T a3f8d6ac7d5c4cc2b1b795c908d81b60 P caf932fb540e4ddd8a78ad228da6c97a [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:36:59.743373 20620 raft_consensus.cc:504] T a3f8d6ac7d5c4cc2b1b795c908d81b60 P caf932fb540e4ddd8a78ad228da6c97a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "330aa80ec1ea4f0785a33dfe4b1c916f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36627 } } peers { permanent_uuid: "cbcf3fa3797145778c907290027e807b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38255 } } peers { permanent_uuid: "caf932fb540e4ddd8a78ad228da6c97a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46047 } }
I20260925 16:36:59.743804 20620 leader_election.cc:288] T a3f8d6ac7d5c4cc2b1b795c908d81b60 P caf932fb540e4ddd8a78ad228da6c97a [CANDIDATE]: Term 1 election: Requested vote from peers 330aa80ec1ea4f0785a33dfe4b1c916f (127.0.98.2:36627), cbcf3fa3797145778c907290027e807b (127.0.98.1:38255)
I20260925 16:36:59.744182 20289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3f8d6ac7d5c4cc2b1b795c908d81b60" candidate_uuid: "caf932fb540e4ddd8a78ad228da6c97a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbcf3fa3797145778c907290027e807b"
I20260925 16:36:59.744172 20420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3f8d6ac7d5c4cc2b1b795c908d81b60" candidate_uuid: "caf932fb540e4ddd8a78ad228da6c97a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "330aa80ec1ea4f0785a33dfe4b1c916f"
I20260925 16:36:59.744304 20289 raft_consensus.cc:3037] T a3f8d6ac7d5c4cc2b1b795c908d81b60 P cbcf3fa3797145778c907290027e807b [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:36:59.744323 20420 raft_consensus.cc:3037] T a3f8d6ac7d5c4cc2b1b795c908d81b60 P 330aa80ec1ea4f0785a33dfe4b1c916f [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:36:59.745276 20289 raft_consensus.cc:2454] T a3f8d6ac7d5c4cc2b1b795c908d81b60 P cbcf3fa3797145778c907290027e807b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate caf932fb540e4ddd8a78ad228da6c97a in term 1.
I20260925 16:36:59.745405 20420 raft_consensus.cc:2454] T a3f8d6ac7d5c4cc2b1b795c908d81b60 P 330aa80ec1ea4f0785a33dfe4b1c916f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate caf932fb540e4ddd8a78ad228da6c97a in term 1.
I20260925 16:36:59.745615 20484 leader_election.cc:302] T a3f8d6ac7d5c4cc2b1b795c908d81b60 P caf932fb540e4ddd8a78ad228da6c97a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: caf932fb540e4ddd8a78ad228da6c97a, cbcf3fa3797145778c907290027e807b; no voters:
I20260925 16:36:59.745781 20620 raft_consensus.cc:2781] T a3f8d6ac7d5c4cc2b1b795c908d81b60 P caf932fb540e4ddd8a78ad228da6c97a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:36:59.745983 20620 raft_consensus.cc:686] T a3f8d6ac7d5c4cc2b1b795c908d81b60 P caf932fb540e4ddd8a78ad228da6c97a [term 1 LEADER]: Becoming Leader. State: Replica: caf932fb540e4ddd8a78ad228da6c97a, State: Running, Role: LEADER
I20260925 16:36:59.746318 20620 consensus_queue.cc:237] T a3f8d6ac7d5c4cc2b1b795c908d81b60 P caf932fb540e4ddd8a78ad228da6c97a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "330aa80ec1ea4f0785a33dfe4b1c916f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36627 } } peers { permanent_uuid: "cbcf3fa3797145778c907290027e807b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38255 } } peers { permanent_uuid: "caf932fb540e4ddd8a78ad228da6c97a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46047 } }
I20260925 16:36:59.749354 20146 catalog_manager.cc:5495] T a3f8d6ac7d5c4cc2b1b795c908d81b60 P caf932fb540e4ddd8a78ad228da6c97a reported cstate change: term changed from 0 to 1, leader changed from <none> to caf932fb540e4ddd8a78ad228da6c97a (127.0.98.3). New cstate: current_term: 1 leader_uuid: "caf932fb540e4ddd8a78ad228da6c97a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "330aa80ec1ea4f0785a33dfe4b1c916f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cbcf3fa3797145778c907290027e807b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "caf932fb540e4ddd8a78ad228da6c97a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46047 } health_report { overall_health: HEALTHY } } }
I20260925 16:36:59.789189 20613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:59.785904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55452 (local address 127.0.98.3:46047)
0925 16:36:59.786032 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 16:36:59.786036 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:36:59.786049 (+ 13us) server_negotiation.cc:407] Connection header received
0925 16:36:59.786148 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:36:59.786150 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:36:59.786203 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:36:59.786290 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:36:59.786494 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:59.787041 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:59.787890 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:36:59.788088 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:36:59.788159 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:36:59.788399 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:36:59.788801 (+ 402us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:36:59.788899 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 16:36:59.788949 (+ 50us) server_negotiation.cc:299] Negotiation successful
0925 16:36:59.788989 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 16:36:59.795030 20420 raft_consensus.cc:1261] T a3f8d6ac7d5c4cc2b1b795c908d81b60 P 330aa80ec1ea4f0785a33dfe4b1c916f [term 1 FOLLOWER]: Refusing update from remote peer caf932fb540e4ddd8a78ad228da6c97a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:36:59.795030 20289 raft_consensus.cc:1261] T a3f8d6ac7d5c4cc2b1b795c908d81b60 P cbcf3fa3797145778c907290027e807b [term 1 FOLLOWER]: Refusing update from remote peer caf932fb540e4ddd8a78ad228da6c97a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:36:59.795725 20620 consensus_queue.cc:1035] T a3f8d6ac7d5c4cc2b1b795c908d81b60 P caf932fb540e4ddd8a78ad228da6c97a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbcf3fa3797145778c907290027e807b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:36:59.795921 20624 consensus_queue.cc:1035] T a3f8d6ac7d5c4cc2b1b795c908d81b60 P caf932fb540e4ddd8a78ad228da6c97a [LEADER]: Connected to new peer: Peer: permanent_uuid: "330aa80ec1ea4f0785a33dfe4b1c916f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:36:59.802678 20628 mvcc.cc:205] Tried to move back new op lower bound from 7333290884273614848 to 7333290884083490816. Current Snapshot: MvccSnapshot[applied={T|T < 7333290884273614848}]
I20260925 16:36:59.803817 20627 mvcc.cc:205] Tried to move back new op lower bound from 7333290884273614848 to 7333290884083490816. Current Snapshot: MvccSnapshot[applied={T|T < 7333290884273614848}]
I20260925 16:36:59.804405 20629 mvcc.cc:205] Tried to move back new op lower bound from 7333290884273614848 to 7333290884083490816. Current Snapshot: MvccSnapshot[applied={T|T < 7333290884273614848}]
I20260925 16:36:59.813488 20145 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35406:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:36:59.813668 20145 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35406:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:36:59.816186 20145 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5d8bedda64854c23846f54d85ce8bbdf: Sending DeleteTablet for 3 replicas of tablet a3f8d6ac7d5c4cc2b1b795c908d81b60
I20260925 16:36:59.816953 20400 tablet_service.cc:1514] Processing DeleteTablet for tablet a3f8d6ac7d5c4cc2b1b795c908d81b60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:36:59 UTC) from {username='slave'} at 127.0.0.1:34646
I20260925 16:36:59.817112 20269 tablet_service.cc:1514] Processing DeleteTablet for tablet a3f8d6ac7d5c4cc2b1b795c908d81b60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:36:59 UTC) from {username='slave'} at 127.0.0.1:58194
I20260925 16:36:59.817126 20531 tablet_service.cc:1514] Processing DeleteTablet for tablet a3f8d6ac7d5c4cc2b1b795c908d81b60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:36:59 UTC) from {username='slave'} at 127.0.0.1:55450
I20260925 16:36:59.817538 20637 tablet_replica.cc:330] T a3f8d6ac7d5c4cc2b1b795c908d81b60 P cbcf3fa3797145778c907290027e807b: stopping tablet replica
I20260925 16:36:59.817833 20638 tablet_replica.cc:330] T a3f8d6ac7d5c4cc2b1b795c908d81b60 P caf932fb540e4ddd8a78ad228da6c97a: stopping tablet replica
I20260925 16:36:59.817894 20637 raft_consensus.cc:2229] T a3f8d6ac7d5c4cc2b1b795c908d81b60 P cbcf3fa3797145778c907290027e807b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:36:59.818073 20638 raft_consensus.cc:2229] T a3f8d6ac7d5c4cc2b1b795c908d81b60 P caf932fb540e4ddd8a78ad228da6c97a [term 1 LEADER]: Raft consensus shutting down.
I20260925 16:36:59.818194 20637 raft_consensus.cc:2258] T a3f8d6ac7d5c4cc2b1b795c908d81b60 P cbcf3fa3797145778c907290027e807b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:36:59.818210 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 20207
I20260925 16:36:59.820267 20638 raft_consensus.cc:2258] T a3f8d6ac7d5c4cc2b1b795c908d81b60 P caf932fb540e4ddd8a78ad228da6c97a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:36:59.821492 20638 ts_tablet_manager.cc:1905] T a3f8d6ac7d5c4cc2b1b795c908d81b60 P caf932fb540e4ddd8a78ad228da6c97a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:36:59.822824 20636 tablet_replica.cc:330] T a3f8d6ac7d5c4cc2b1b795c908d81b60 P 330aa80ec1ea4f0785a33dfe4b1c916f: stopping tablet replica
I20260925 16:36:59.823192 20636 raft_consensus.cc:2229] T a3f8d6ac7d5c4cc2b1b795c908d81b60 P 330aa80ec1ea4f0785a33dfe4b1c916f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:36:59.823539 20636 raft_consensus.cc:2258] T a3f8d6ac7d5c4cc2b1b795c908d81b60 P 330aa80ec1ea4f0785a33dfe4b1c916f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:36:59.824056 20638 ts_tablet_manager.cc:1918] T a3f8d6ac7d5c4cc2b1b795c908d81b60 P caf932fb540e4ddd8a78ad228da6c97a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:36:59.824143 20638 log.cc:1196] T a3f8d6ac7d5c4cc2b1b795c908d81b60 P caf932fb540e4ddd8a78ad228da6c97a: Deleting WAL directory at /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353966814071-392-0/minicluster-data/ts-2/wal/wals/a3f8d6ac7d5c4cc2b1b795c908d81b60
I20260925 16:36:59.824455 20638 ts_tablet_manager.cc:1939] T a3f8d6ac7d5c4cc2b1b795c908d81b60 P caf932fb540e4ddd8a78ad228da6c97a: Deleting consensus metadata
I20260925 16:36:59.825266 20636 ts_tablet_manager.cc:1905] T a3f8d6ac7d5c4cc2b1b795c908d81b60 P 330aa80ec1ea4f0785a33dfe4b1c916f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:36:59.825807 20134 catalog_manager.cc:4841] TS caf932fb540e4ddd8a78ad228da6c97a (127.0.98.3:46047): tablet a3f8d6ac7d5c4cc2b1b795c908d81b60 (table test-table [id=88903894c57d4821acfc0dbb2b87f958]) successfully deleted
W20260925 16:36:59.825814 20131 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38255 (error 108)
I20260925 16:36:59.826179 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 20338
I20260925 16:36:59.831736 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 20469
I20260925 16:36:59.833146 20608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:59.831978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38255 (local address 127.0.0.1:58208)
0925 16:36:59.833010 (+ 1032us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:59.833072 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":951}
W20260925 16:36:59.833329 20131 catalog_manager.cc:4584] TS cbcf3fa3797145778c907290027e807b (127.0.98.1:38255): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a3f8d6ac7d5c4cc2b1b795c908d81b60: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38255: connect: Connection refused (error 111)
I20260925 16:36:59.834177 20608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:36:59.833886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36627 (local address 127.0.0.1:34662)
0925 16:36:59.834032 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 16:36:59.834094 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:36627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260925 16:36:59.837646 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 20116
2026-09-25T16:36:59Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1060 ms)
[----------] 6 tests from EncryptionPolicyTest (10343 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790353966814071-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:36:59 dist-test-slave-65pf krb5kdc[20642](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:36:59 dist-test-slave-65pf krb5kdc[20642](info): set up 2 sockets
Sep 25 16:36:59 dist-test-slave-65pf krb5kdc[20642](info): commencing operation
krb5kdc: starting...
W20260925 16:37:00.867053 392 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790353966814071-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790353966814071-392-0/krb5kdc/test-user.keytab.
Sep 25 16:37:00 dist-test-slave-65pf krb5kdc[20642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:37:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:37:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790353966814071-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790353966814071-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 16:37:00.989240 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790353966814071-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790353966814071-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790353966814071-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35889
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790353966814071-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35889
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:37:01.097932 20655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:37:01.098218 20655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:37:01.098260 20655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:37:01.102418 20655 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:37:01.102489 20655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:37:01.102509 20655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:37:01.102526 20655 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:37:01.102541 20655 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:37:01.108470 20655 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790353966814071-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790353966814071-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35889
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790353966814071-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790353966814071-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:37:01.109629 20655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:37:01.110780 20655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:37:01.119324 20660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:37:01.119314 20661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:37:01.119433 20655 server_base.cc:1034] running on GCE node
W20260925 16:37:01.119315 20663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:37:01.120357 20655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:37:01.121536 20655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:37:01.122653 20655 hybrid_clock.cc:648] HybridClock initialized: now 1790354221122623 us; error 39 us; skew 500 ppm
Sep 25 16:37:01 dist-test-slave-65pf krb5kdc[20642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354221, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:37:01.125530 20655 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 16:37:01.126492 20655 webserver.cc:460] Webserver started at http://127.0.0.1:36153/ using document root <none> and password file <none>
I20260925 16:37:01.127096 20655 fs_manager.cc:359] Metadata directory not provided
I20260925 16:37:01.127157 20655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:37:01.127388 20655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:37:01.129392 20655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790353966814071-392-0/minicluster-data/master-0/data/instance:
uuid: "8b38d14d3618401dbaa73af2f7bbdcb5"
format_stamp: "Formatted at 2026-09-25 16:37:01 on dist-test-slave-65pf"
server_key: "9c3cbc5636e0945a7cd9afdeec0d3e34"
server_key_iv: "fb99357c17cde7649a7ec698881ef6e2"
server_key_version: "encryptionkey@0"
I20260925 16:37:01.129961 20655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790353966814071-392-0/minicluster-data/master-0/wal/instance:
uuid: "8b38d14d3618401dbaa73af2f7bbdcb5"
format_stamp: "Formatted at 2026-09-25 16:37:01 on dist-test-slave-65pf"
server_key: "9c3cbc5636e0945a7cd9afdeec0d3e34"
server_key_iv: "fb99357c17cde7649a7ec698881ef6e2"
server_key_version: "encryptionkey@0"
I20260925 16:37:01.133737 20655 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 16:37:01.136111 20670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:37:01.137274 20655 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 16:37:01.137416 20655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790353966814071-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790353966814071-392-0/minicluster-data/master-0/wal
uuid: "8b38d14d3618401dbaa73af2f7bbdcb5"
format_stamp: "Formatted at 2026-09-25 16:37:01 on dist-test-slave-65pf"
server_key: "9c3cbc5636e0945a7cd9afdeec0d3e34"
server_key_iv: "fb99357c17cde7649a7ec698881ef6e2"
server_key_version: "encryptionkey@0"
I20260925 16:37:01.137530 20655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790353966814071-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790353966814071-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790353966814071-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:37:01.151897 20655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:37:01.154747 20655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:37:01.154958 20655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:37:01.162092 20655 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35889
I20260925 16:37:01.162112 20722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35889 every 8 connection(s)
I20260925 16:37:01.163180 20655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:37:01.165699 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 20655
I20260925 16:37:01.165849 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790353966814071-392-0/minicluster-data/master-0/wal/instance
I20260925 16:37:01.166160 392 external_mini_cluster.cc:1428] Setting key b616967c1ccabe7056f385f4c627141e
I20260925 16:37:01.166253 20723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:37:01.172506 20723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b38d14d3618401dbaa73af2f7bbdcb5: Bootstrap starting.
I20260925 16:37:01.174998 20723 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b38d14d3618401dbaa73af2f7bbdcb5: Neither blocks nor log segments found. Creating new log.
Sep 25 16:37:01 dist-test-slave-65pf krb5kdc[20642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 16:37:01.175911 20723 log.cc:824] T 00000000000000000000000000000000 P 8b38d14d3618401dbaa73af2f7bbdcb5: Log is configured to *not* fsync() on all Append() calls
I20260925 16:37:01.178071 20723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b38d14d3618401dbaa73af2f7bbdcb5: No bootstrap required, opened a new log
I20260925 16:37:01.180562 20726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:01.167637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41053 (local address 127.0.0.1:35889)
0925 16:37:01.168064 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:01.168079 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:01.168113 (+ 34us) server_negotiation.cc:407] Connection header received
0925 16:37:01.168963 (+ 850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:01.168981 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:01.169320 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:01.169665 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:01.170461 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:01.171516 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:01.172384 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:01.172924 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:01.173099 (+ 175us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:01.175867 (+ 2768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:37:01.175873 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:37:01.175891 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:37:01.175921 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:37:01.177692 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:01.178262 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:01.178266 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:01.178270 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:01.178384 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:01.178729 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:01.178731 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:01.178732 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:01.179111 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:37:01.179262 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:01.179584 (+ 322us) server_negotiation.cc:299] Negotiation successful
0925 16:37:01.179816 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":138,"threads_started":1}
I20260925 16:37:01.181972 20723 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b38d14d3618401dbaa73af2f7bbdcb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b38d14d3618401dbaa73af2f7bbdcb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35889 } }
I20260925 16:37:01.182236 20723 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b38d14d3618401dbaa73af2f7bbdcb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:37:01.182299 20723 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b38d14d3618401dbaa73af2f7bbdcb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b38d14d3618401dbaa73af2f7bbdcb5, State: Initialized, Role: FOLLOWER
I20260925 16:37:01.182806 20723 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b38d14d3618401dbaa73af2f7bbdcb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b38d14d3618401dbaa73af2f7bbdcb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35889 } }
I20260925 16:37:01.182965 20723 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b38d14d3618401dbaa73af2f7bbdcb5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:37:01.183068 20723 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b38d14d3618401dbaa73af2f7bbdcb5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:37:01.183168 20723 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b38d14d3618401dbaa73af2f7bbdcb5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:37:01.183578 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:37:01.184267 20723 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b38d14d3618401dbaa73af2f7bbdcb5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b38d14d3618401dbaa73af2f7bbdcb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35889 } }
I20260925 16:37:01.184625 20723 leader_election.cc:302] T 00000000000000000000000000000000 P 8b38d14d3618401dbaa73af2f7bbdcb5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b38d14d3618401dbaa73af2f7bbdcb5; no voters:
I20260925 16:37:01.184957 20723 leader_election.cc:288] T 00000000000000000000000000000000 P 8b38d14d3618401dbaa73af2f7bbdcb5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:37:01.185109 20730 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b38d14d3618401dbaa73af2f7bbdcb5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:37:01.185345 20730 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b38d14d3618401dbaa73af2f7bbdcb5 [term 1 LEADER]: Becoming Leader. State: Replica: 8b38d14d3618401dbaa73af2f7bbdcb5, State: Running, Role: LEADER
I20260925 16:37:01.185671 20730 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b38d14d3618401dbaa73af2f7bbdcb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b38d14d3618401dbaa73af2f7bbdcb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35889 } }
I20260925 16:37:01.186312 20723 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b38d14d3618401dbaa73af2f7bbdcb5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:37:01.186906 20730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b38d14d3618401dbaa73af2f7bbdcb5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b38d14d3618401dbaa73af2f7bbdcb5. Latest consensus state: current_term: 1 leader_uuid: "8b38d14d3618401dbaa73af2f7bbdcb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b38d14d3618401dbaa73af2f7bbdcb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35889 } } }
I20260925 16:37:01.187024 20730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b38d14d3618401dbaa73af2f7bbdcb5 [sys.catalog]: This master's current role is: LEADER
I20260925 16:37:01.187336 20733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b38d14d3618401dbaa73af2f7bbdcb5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b38d14d3618401dbaa73af2f7bbdcb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b38d14d3618401dbaa73af2f7bbdcb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35889 } } }
I20260925 16:37:01.187419 20733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b38d14d3618401dbaa73af2f7bbdcb5 [sys.catalog]: This master's current role is: LEADER
W20260925 16:37:01.191354 20748 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8b38d14d3618401dbaa73af2f7bbdcb5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 16:37:01.191450 20748 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 16:37:01.191529 20738 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:37:01.192341 20738 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:37:01.194712 20726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:01.185771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36697 (local address 127.0.0.1:35889)
0925 16:37:01.185910 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:01.185915 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:01.186105 (+ 190us) server_negotiation.cc:407] Connection header received
0925 16:37:01.186241 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:01.186244 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:01.186289 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:01.186369 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:01.187259 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:01.188076 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:01.189340 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:01.189578 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:01.189636 (+ 58us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:01.191260 (+ 1624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:37:01.191264 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:37:01.191267 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:37:01.191291 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:37:01.192906 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:01.193374 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:01.193378 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:01.193380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:01.193464 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:01.193783 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:01.193786 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:01.193788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:01.193971 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:37:01.194121 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:01.194412 (+ 291us) server_negotiation.cc:299] Negotiation successful
0925 16:37:01.194519 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 16:37:01.198290 20738 catalog_manager.cc:1347] Generated new cluster ID: b349d80f52374f35b43259acde733473
I20260925 16:37:01.198361 20738 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:37:01.208935 20738 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:37:01.209882 20738 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:37:01.222782 20738 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b38d14d3618401dbaa73af2f7bbdcb5: Generated new TSK 0
I20260925 16:37:01.223542 20738 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:37:01.231699 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 20655
W20260925 16:37:01.236490 20640 connection.cc:535] client connection to 127.0.0.1:35889 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35889): Connection reset by peer (error 104)
2026-09-25T16:37:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1391 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790353966814071-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:37:01 dist-test-slave-65pf krb5kdc[20754](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:37:01 dist-test-slave-65pf krb5kdc[20754](info): set up 2 sockets
Sep 25 16:37:01 dist-test-slave-65pf krb5kdc[20754](info): commencing operation
krb5kdc: starting...
W20260925 16:37:02.275020 392 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790353966814071-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790353966814071-392-0/krb5kdc/test-user.keytab.
Sep 25 16:37:02 dist-test-slave-65pf krb5kdc[20754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:37:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:37:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790353966814071-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790353966814071-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 16:37:02.401273 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790353966814071-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790353966814071-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790353966814071-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38347
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790353966814071-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38347
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:37:02.510548 20767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:37:02.510931 20767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:37:02.510974 20767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:37:02.514976 20767 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:37:02.515033 20767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:37:02.515053 20767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:37:02.515069 20767 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:37:02.515084 20767 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:37:02.520397 20767 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790353966814071-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790353966814071-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38347
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790353966814071-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790353966814071-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:37:02.521392 20767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:37:02.522346 20767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:37:02.530648 20775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:37:02.530892 20767 server_base.cc:1034] running on GCE node
W20260925 16:37:02.530655 20773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:37:02.530655 20772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:37:02.531607 20767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:37:02.532598 20767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:37:02.533780 20767 hybrid_clock.cc:648] HybridClock initialized: now 1790354222533746 us; error 48 us; skew 500 ppm
Sep 25 16:37:02 dist-test-slave-65pf krb5kdc[20754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354222, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:37:02.536368 20767 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 16:37:02.537278 20767 webserver.cc:460] Webserver started at http://127.0.0.1:36515/ using document root <none> and password file <none>
I20260925 16:37:02.537776 20767 fs_manager.cc:359] Metadata directory not provided
I20260925 16:37:02.537829 20767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:37:02.537988 20767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:37:02.539943 20767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790353966814071-392-0/minicluster-data/master-0/data/instance:
uuid: "1031ebd3227a4c16ac3e63dc5c95ce33"
format_stamp: "Formatted at 2026-09-25 16:37:02 on dist-test-slave-65pf"
server_key: "0f327c7d49894c5e6b3f98d2d3a1b007"
server_key_iv: "4a28c6d04e66c908ac729836bc0cf994"
server_key_version: "encryptionkey@0"
I20260925 16:37:02.540443 20767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790353966814071-392-0/minicluster-data/master-0/wal/instance:
uuid: "1031ebd3227a4c16ac3e63dc5c95ce33"
format_stamp: "Formatted at 2026-09-25 16:37:02 on dist-test-slave-65pf"
server_key: "0f327c7d49894c5e6b3f98d2d3a1b007"
server_key_iv: "4a28c6d04e66c908ac729836bc0cf994"
server_key_version: "encryptionkey@0"
I20260925 16:37:02.543889 20767 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 16:37:02.546077 20782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:37:02.547429 20767 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:37:02.547528 20767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790353966814071-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790353966814071-392-0/minicluster-data/master-0/wal
uuid: "1031ebd3227a4c16ac3e63dc5c95ce33"
format_stamp: "Formatted at 2026-09-25 16:37:02 on dist-test-slave-65pf"
server_key: "0f327c7d49894c5e6b3f98d2d3a1b007"
server_key_iv: "4a28c6d04e66c908ac729836bc0cf994"
server_key_version: "encryptionkey@0"
I20260925 16:37:02.547609 20767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790353966814071-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790353966814071-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790353966814071-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:37:02.572525 20767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:37:02.575222 20767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:37:02.575380 20767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:37:02.582302 20767 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38347
I20260925 16:37:02.582320 20834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38347 every 8 connection(s)
I20260925 16:37:02.583340 20767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:37:02.586017 20835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:37:02.587845 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 20767
I20260925 16:37:02.588037 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790353966814071-392-0/minicluster-data/master-0/wal/instance
I20260925 16:37:02.588326 392 external_mini_cluster.cc:1428] Setting key 2518565763a366744115b2f8f98b9a2d
I20260925 16:37:02.592411 20835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1031ebd3227a4c16ac3e63dc5c95ce33: Bootstrap starting.
I20260925 16:37:02.594471 20835 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1031ebd3227a4c16ac3e63dc5c95ce33: Neither blocks nor log segments found. Creating new log.
I20260925 16:37:02.595381 20835 log.cc:824] T 00000000000000000000000000000000 P 1031ebd3227a4c16ac3e63dc5c95ce33: Log is configured to *not* fsync() on all Append() calls
I20260925 16:37:02.596944 20838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:02.589716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52063 (local address 127.0.0.1:38347)
0925 16:37:02.590135 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:02.590150 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:02.590185 (+ 35us) server_negotiation.cc:407] Connection header received
0925 16:37:02.590923 (+ 738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:02.590943 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:02.591292 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:02.591628 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:02.592032 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:02.593002 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:02.593824 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:02.594248 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:02.594416 (+ 168us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:02.595322 (+ 906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:37:02.595327 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:37:02.595339 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:37:02.595755 (+ 416us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:37:02.595806 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:37:02.595828 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:37:02.595963 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:02.596135 (+ 172us) server_negotiation.cc:299] Negotiation successful
0925 16:37:02.596269 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":128,"threads_started":1}
I20260925 16:37:02.597337 20835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1031ebd3227a4c16ac3e63dc5c95ce33: No bootstrap required, opened a new log
I20260925 16:37:02.599917 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:37:02.601437 20835 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1031ebd3227a4c16ac3e63dc5c95ce33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1031ebd3227a4c16ac3e63dc5c95ce33" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38347 } }
I20260925 16:37:02.601662 20835 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1031ebd3227a4c16ac3e63dc5c95ce33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:37:02.601744 20835 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1031ebd3227a4c16ac3e63dc5c95ce33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1031ebd3227a4c16ac3e63dc5c95ce33, State: Initialized, Role: FOLLOWER
I20260925 16:37:02.602177 20835 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1031ebd3227a4c16ac3e63dc5c95ce33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1031ebd3227a4c16ac3e63dc5c95ce33" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38347 } }
I20260925 16:37:02.602320 20835 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1031ebd3227a4c16ac3e63dc5c95ce33 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:37:02.602387 20835 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1031ebd3227a4c16ac3e63dc5c95ce33 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:37:02.602488 20835 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1031ebd3227a4c16ac3e63dc5c95ce33 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:37:02.603631 20835 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1031ebd3227a4c16ac3e63dc5c95ce33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1031ebd3227a4c16ac3e63dc5c95ce33" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38347 } }
I20260925 16:37:02.603981 20835 leader_election.cc:302] T 00000000000000000000000000000000 P 1031ebd3227a4c16ac3e63dc5c95ce33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1031ebd3227a4c16ac3e63dc5c95ce33; no voters:
I20260925 16:37:02.604295 20835 leader_election.cc:288] T 00000000000000000000000000000000 P 1031ebd3227a4c16ac3e63dc5c95ce33 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:37:02.604413 20845 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1031ebd3227a4c16ac3e63dc5c95ce33 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:37:02.604624 20845 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1031ebd3227a4c16ac3e63dc5c95ce33 [term 1 LEADER]: Becoming Leader. State: Replica: 1031ebd3227a4c16ac3e63dc5c95ce33, State: Running, Role: LEADER
I20260925 16:37:02.605001 20845 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1031ebd3227a4c16ac3e63dc5c95ce33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1031ebd3227a4c16ac3e63dc5c95ce33" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38347 } }
I20260925 16:37:02.605521 20835 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1031ebd3227a4c16ac3e63dc5c95ce33 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:37:02.606700 20847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1031ebd3227a4c16ac3e63dc5c95ce33 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1031ebd3227a4c16ac3e63dc5c95ce33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1031ebd3227a4c16ac3e63dc5c95ce33" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38347 } } }
I20260925 16:37:02.606843 20847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1031ebd3227a4c16ac3e63dc5c95ce33 [sys.catalog]: This master's current role is: LEADER
I20260925 16:37:02.607076 20838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:02.601918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41857 (local address 127.0.0.1:38347)
0925 16:37:02.602115 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:02.602118 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:02.602301 (+ 183us) server_negotiation.cc:407] Connection header received
0925 16:37:02.602458 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:02.602462 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:02.602511 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:02.602624 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:02.602885 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:02.603639 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:02.604700 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:02.605015 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:02.605099 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:02.606209 (+ 1110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:37:02.606213 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:37:02.606216 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:37:02.606231 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:37:02.606258 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:37:02.606271 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:37:02.606356 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:02.606878 (+ 522us) server_negotiation.cc:299] Negotiation successful
0925 16:37:02.606911 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136}
I20260925 16:37:02.607271 20846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1031ebd3227a4c16ac3e63dc5c95ce33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1031ebd3227a4c16ac3e63dc5c95ce33. Latest consensus state: current_term: 1 leader_uuid: "1031ebd3227a4c16ac3e63dc5c95ce33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1031ebd3227a4c16ac3e63dc5c95ce33" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38347 } } }
I20260925 16:37:02.607347 20846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1031ebd3227a4c16ac3e63dc5c95ce33 [sys.catalog]: This master's current role is: LEADER
W20260925 16:37:02.610019 20860 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1031ebd3227a4c16ac3e63dc5c95ce33: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 16:37:02.610080 20860 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 16:37:02.610374 20854 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:37:02.611148 20854 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:37:02.616042 20854 catalog_manager.cc:1347] Generated new cluster ID: a7f25455d1704c64a2bc0fc2f5b81fe6
I20260925 16:37:02.616099 20854 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:37:02.629446 20854 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:37:02.631287 20854 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:37:02.636677 20854 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1031ebd3227a4c16ac3e63dc5c95ce33: Generated new TSK 0
I20260925 16:37:02.637384 20854 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:37:02.642155 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 20767
W20260925 16:37:02.647501 20752 connection.cc:535] client connection to 127.0.0.1:38347 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38347): Connection reset by peer (error 104)
2026-09-25T16:37:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1411 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790353966814071-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:37:02 dist-test-slave-65pf krb5kdc[20866](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:37:02 dist-test-slave-65pf krb5kdc[20866](info): set up 2 sockets
Sep 25 16:37:02 dist-test-slave-65pf krb5kdc[20866](info): commencing operation
krb5kdc: starting...
W20260925 16:37:03.670835 392 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790353966814071-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790353966814071-392-0/krb5kdc/test-user.keytab.
Sep 25 16:37:03 dist-test-slave-65pf krb5kdc[20866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:37:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:37:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790353966814071-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790353966814071-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 16:37:03.794078 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790353966814071-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790353966814071-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790353966814071-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40657
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790353966814071-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40657
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:37:03.905475 20879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:37:03.905778 20879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:37:03.905823 20879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:37:03.909965 20879 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:37:03.910037 20879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:37:03.910058 20879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:37:03.910130 20879 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:37:03.910161 20879 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:37:03.915611 20879 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790353966814071-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790353966814071-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40657
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790353966814071-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790353966814071-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:37:03.916754 20879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:37:03.917845 20879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:37:03.926689 20885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:37:03.926735 20887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:37:03.926717 20884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:37:03.927021 20879 server_base.cc:1034] running on GCE node
I20260925 16:37:03.927603 20879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:37:03.928714 20879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:37:03.929940 20879 hybrid_clock.cc:648] HybridClock initialized: now 1790354223929908 us; error 48 us; skew 500 ppm
Sep 25 16:37:03 dist-test-slave-65pf krb5kdc[20866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354223, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:37:03.932857 20879 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 16:37:03.934046 20879 webserver.cc:460] Webserver started at http://127.0.0.1:33049/ using document root <none> and password file <none>
I20260925 16:37:03.934705 20879 fs_manager.cc:359] Metadata directory not provided
I20260925 16:37:03.934792 20879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:37:03.935009 20879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:37:03.937078 20879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790353966814071-392-0/minicluster-data/master-0/data/instance:
uuid: "788a0606e182465587b77cb0c11a164d"
format_stamp: "Formatted at 2026-09-25 16:37:03 on dist-test-slave-65pf"
server_key: "c9fb2ec2b9f8a954572a571d93c10714"
server_key_iv: "42e6b12d0d1f1ea09e7de24069f39f33"
server_key_version: "encryptionkey@0"
I20260925 16:37:03.937642 20879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790353966814071-392-0/minicluster-data/master-0/wal/instance:
uuid: "788a0606e182465587b77cb0c11a164d"
format_stamp: "Formatted at 2026-09-25 16:37:03 on dist-test-slave-65pf"
server_key: "c9fb2ec2b9f8a954572a571d93c10714"
server_key_iv: "42e6b12d0d1f1ea09e7de24069f39f33"
server_key_version: "encryptionkey@0"
I20260925 16:37:03.941517 20879 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 16:37:03.943979 20894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:37:03.945250 20879 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 16:37:03.945398 20879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790353966814071-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790353966814071-392-0/minicluster-data/master-0/wal
uuid: "788a0606e182465587b77cb0c11a164d"
format_stamp: "Formatted at 2026-09-25 16:37:03 on dist-test-slave-65pf"
server_key: "c9fb2ec2b9f8a954572a571d93c10714"
server_key_iv: "42e6b12d0d1f1ea09e7de24069f39f33"
server_key_version: "encryptionkey@0"
I20260925 16:37:03.945531 20879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790353966814071-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790353966814071-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790353966814071-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:37:03.950142 20879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:37:03.953413 20879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:37:03.953611 20879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:37:03.960768 20879 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40657
I20260925 16:37:03.960794 20946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40657 every 8 connection(s)
I20260925 16:37:03.961732 20879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:37:03.964720 20947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:37:03.970119 20947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 788a0606e182465587b77cb0c11a164d: Bootstrap starting.
I20260925 16:37:03.970460 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 20879
I20260925 16:37:03.970630 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790353966814071-392-0/minicluster-data/master-0/wal/instance
I20260925 16:37:03.970918 392 external_mini_cluster.cc:1428] Setting key e3d104e893d2837e7d007d37b9eb2d3e
I20260925 16:37:03.973055 20947 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 788a0606e182465587b77cb0c11a164d: Neither blocks nor log segments found. Creating new log.
I20260925 16:37:03.973799 20947 log.cc:824] T 00000000000000000000000000000000 P 788a0606e182465587b77cb0c11a164d: Log is configured to *not* fsync() on all Append() calls
I20260925 16:37:03.976166 20947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 788a0606e182465587b77cb0c11a164d: No bootstrap required, opened a new log
I20260925 16:37:03.976590 20950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:03.972360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48327 (local address 127.0.0.1:40657)
0925 16:37:03.972888 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:03.972904 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:03.972948 (+ 44us) server_negotiation.cc:407] Connection header received
0925 16:37:03.973684 (+ 736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:03.973710 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:03.974055 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:03.974455 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:03.975246 (+ 791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:37:03.975250 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:37:03.975262 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:37:03.975271 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:37:03.975322 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:37:03.975342 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:37:03.975440 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:03.975584 (+ 144us) server_negotiation.cc:299] Negotiation successful
0925 16:37:03.975805 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":141,"threads_started":1}
I20260925 16:37:03.979594 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:37:03.980271 20947 raft_consensus.cc:348] T 00000000000000000000000000000000 P 788a0606e182465587b77cb0c11a164d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788a0606e182465587b77cb0c11a164d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40657 } }
I20260925 16:37:03.980513 20947 raft_consensus.cc:374] T 00000000000000000000000000000000 P 788a0606e182465587b77cb0c11a164d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:37:03.980588 20947 raft_consensus.cc:729] T 00000000000000000000000000000000 P 788a0606e182465587b77cb0c11a164d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 788a0606e182465587b77cb0c11a164d, State: Initialized, Role: FOLLOWER
I20260925 16:37:03.981153 20947 consensus_queue.cc:260] T 00000000000000000000000000000000 P 788a0606e182465587b77cb0c11a164d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788a0606e182465587b77cb0c11a164d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40657 } }
I20260925 16:37:03.981328 20947 raft_consensus.cc:388] T 00000000000000000000000000000000 P 788a0606e182465587b77cb0c11a164d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:37:03.981417 20947 raft_consensus.cc:482] T 00000000000000000000000000000000 P 788a0606e182465587b77cb0c11a164d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:37:03.981514 20947 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 788a0606e182465587b77cb0c11a164d [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:37:03.982544 20947 raft_consensus.cc:504] T 00000000000000000000000000000000 P 788a0606e182465587b77cb0c11a164d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788a0606e182465587b77cb0c11a164d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40657 } }
I20260925 16:37:03.982921 20947 leader_election.cc:302] T 00000000000000000000000000000000 P 788a0606e182465587b77cb0c11a164d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 788a0606e182465587b77cb0c11a164d; no voters:
I20260925 16:37:03.983260 20947 leader_election.cc:288] T 00000000000000000000000000000000 P 788a0606e182465587b77cb0c11a164d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:37:03.983338 20957 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 788a0606e182465587b77cb0c11a164d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:37:03.983531 20957 raft_consensus.cc:686] T 00000000000000000000000000000000 P 788a0606e182465587b77cb0c11a164d [term 1 LEADER]: Becoming Leader. State: Replica: 788a0606e182465587b77cb0c11a164d, State: Running, Role: LEADER
I20260925 16:37:03.983946 20950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:03.981840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42127 (local address 127.0.0.1:40657)
0925 16:37:03.981968 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:03.981974 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:03.982159 (+ 185us) server_negotiation.cc:407] Connection header received
0925 16:37:03.982321 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:03.982324 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:03.982367 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:03.982460 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:03.983280 (+ 820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:37:03.983284 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:37:03.983287 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:37:03.983300 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:37:03.983333 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:37:03.983351 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:37:03.983441 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:03.983769 (+ 328us) server_negotiation.cc:299] Negotiation successful
0925 16:37:03.983837 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 16:37:03.983994 20957 consensus_queue.cc:237] T 00000000000000000000000000000000 P 788a0606e182465587b77cb0c11a164d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788a0606e182465587b77cb0c11a164d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40657 } }
I20260925 16:37:03.984589 20947 sys_catalog.cc:564] T 00000000000000000000000000000000 P 788a0606e182465587b77cb0c11a164d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:37:03.985177 20957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 788a0606e182465587b77cb0c11a164d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 788a0606e182465587b77cb0c11a164d. Latest consensus state: current_term: 1 leader_uuid: "788a0606e182465587b77cb0c11a164d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788a0606e182465587b77cb0c11a164d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40657 } } }
I20260925 16:37:03.985306 20957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 788a0606e182465587b77cb0c11a164d [sys.catalog]: This master's current role is: LEADER
I20260925 16:37:03.985879 20959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 788a0606e182465587b77cb0c11a164d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "788a0606e182465587b77cb0c11a164d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788a0606e182465587b77cb0c11a164d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40657 } } }
I20260925 16:37:03.986099 20959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 788a0606e182465587b77cb0c11a164d [sys.catalog]: This master's current role is: LEADER
I20260925 16:37:03.986308 20966 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:37:03.989631 20966 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:37:03.995322 20966 catalog_manager.cc:1347] Generated new cluster ID: 026438cd8817486fada197bbddadd710
I20260925 16:37:03.995429 20966 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:37:04.013248 20966 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:37:04.013607 20966 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:37:04.023806 20966 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 788a0606e182465587b77cb0c11a164d: Generated new TSK 0
I20260925 16:37:04.024739 20966 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:37:04.039597 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 20879
2026-09-25T16:37:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1397 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790353966814071-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:37:04 dist-test-slave-65pf krb5kdc[20978](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:37:04 dist-test-slave-65pf krb5kdc[20978](info): set up 2 sockets
Sep 25 16:37:04 dist-test-slave-65pf krb5kdc[20978](info): commencing operation
krb5kdc: starting...
W20260925 16:37:05.074972 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790353966814071-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790353966814071-392-0/krb5kdc/test-user.keytab.
Sep 25 16:37:05 dist-test-slave-65pf krb5kdc[20978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:37:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:37:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790353966814071-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790353966814071-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 16:37:05.193651 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790353966814071-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790353966814071-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790353966814071-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42791
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790353966814071-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42791
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:37:05.301373 20991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:37:05.301676 20991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:37:05.301764 20991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:37:05.305861 20991 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:37:05.305953 20991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:37:05.305990 20991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:37:05.306052 20991 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:37:05.306092 20991 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:37:05.311378 20991 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790353966814071-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790353966814071-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42791
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790353966814071-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790353966814071-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:37:05.312449 20991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:37:05.313500 20991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 16:37:05.321975 20991 server_base.cc:1034] running on GCE node
W20260925 16:37:05.321941 20999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:37:05.321965 20996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:37:05.322116 20997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:37:05.322738 20991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:37:05.323750 20991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:37:05.324905 20991 hybrid_clock.cc:648] HybridClock initialized: now 1790354225324878 us; error 40 us; skew 500 ppm
Sep 25 16:37:05 dist-test-slave-65pf krb5kdc[20978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:37:05.332850 20991 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 16:37:05.333701 20991 webserver.cc:460] Webserver started at http://127.0.0.1:39019/ using document root <none> and password file <none>
I20260925 16:37:05.334235 20991 fs_manager.cc:359] Metadata directory not provided
I20260925 16:37:05.334292 20991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:37:05.334463 20991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:37:05.336585 20991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790353966814071-392-0/minicluster-data/master-0/data/instance:
uuid: "611a7b99fbc9433dbb4a4ade10609d2d"
format_stamp: "Formatted at 2026-09-25 16:37:05 on dist-test-slave-65pf"
server_key: "52e345b86cdbc1f954ae7bab84da3a36"
server_key_iv: "e3f8568b364ebf4f8e7159cc6795e18a"
server_key_version: "encryptionkey@0"
I20260925 16:37:05.337083 20991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790353966814071-392-0/minicluster-data/master-0/wal/instance:
uuid: "611a7b99fbc9433dbb4a4ade10609d2d"
format_stamp: "Formatted at 2026-09-25 16:37:05 on dist-test-slave-65pf"
server_key: "52e345b86cdbc1f954ae7bab84da3a36"
server_key_iv: "e3f8568b364ebf4f8e7159cc6795e18a"
server_key_version: "encryptionkey@0"
I20260925 16:37:05.340669 20991 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 16:37:05.342906 21006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:37:05.344081 20991 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 16:37:05.344224 20991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790353966814071-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790353966814071-392-0/minicluster-data/master-0/wal
uuid: "611a7b99fbc9433dbb4a4ade10609d2d"
format_stamp: "Formatted at 2026-09-25 16:37:05 on dist-test-slave-65pf"
server_key: "52e345b86cdbc1f954ae7bab84da3a36"
server_key_iv: "e3f8568b364ebf4f8e7159cc6795e18a"
server_key_version: "encryptionkey@0"
I20260925 16:37:05.344331 20991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790353966814071-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790353966814071-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790353966814071-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:37:05.357987 20991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:37:05.361138 20991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:37:05.361387 20991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:37:05.368285 20991 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42791
I20260925 16:37:05.368307 21058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42791 every 8 connection(s)
I20260925 16:37:05.369259 20991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:37:05.371902 21059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:37:05.377413 21059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 611a7b99fbc9433dbb4a4ade10609d2d: Bootstrap starting.
I20260925 16:37:05.379426 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 20991
I20260925 16:37:05.379580 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790353966814071-392-0/minicluster-data/master-0/wal/instance
I20260925 16:37:05.379859 392 external_mini_cluster.cc:1428] Setting key 78c96f9246f1ebd37e845181aef0101c
I20260925 16:37:05.380020 21059 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 611a7b99fbc9433dbb4a4ade10609d2d: Neither blocks nor log segments found. Creating new log.
I20260925 16:37:05.380991 21059 log.cc:824] T 00000000000000000000000000000000 P 611a7b99fbc9433dbb4a4ade10609d2d: Log is configured to *not* fsync() on all Append() calls
I20260925 16:37:05.383245 21059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 611a7b99fbc9433dbb4a4ade10609d2d: No bootstrap required, opened a new log
I20260925 16:37:05.387564 21059 raft_consensus.cc:348] T 00000000000000000000000000000000 P 611a7b99fbc9433dbb4a4ade10609d2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "611a7b99fbc9433dbb4a4ade10609d2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42791 } }
I20260925 16:37:05.387771 21059 raft_consensus.cc:374] T 00000000000000000000000000000000 P 611a7b99fbc9433dbb4a4ade10609d2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:37:05.387840 21059 raft_consensus.cc:729] T 00000000000000000000000000000000 P 611a7b99fbc9433dbb4a4ade10609d2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 611a7b99fbc9433dbb4a4ade10609d2d, State: Initialized, Role: FOLLOWER
I20260925 16:37:05.388309 21059 consensus_queue.cc:260] T 00000000000000000000000000000000 P 611a7b99fbc9433dbb4a4ade10609d2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "611a7b99fbc9433dbb4a4ade10609d2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42791 } }
I20260925 16:37:05.388435 21059 raft_consensus.cc:388] T 00000000000000000000000000000000 P 611a7b99fbc9433dbb4a4ade10609d2d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:37:05.388518 21059 raft_consensus.cc:482] T 00000000000000000000000000000000 P 611a7b99fbc9433dbb4a4ade10609d2d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:37:05.388618 21059 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 611a7b99fbc9433dbb4a4ade10609d2d [term 0 FOLLOWER]: Advancing to term 1
Sep 25 16:37:05 dist-test-slave-65pf krb5kdc[20978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 16:37:05.389648 21059 raft_consensus.cc:504] T 00000000000000000000000000000000 P 611a7b99fbc9433dbb4a4ade10609d2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "611a7b99fbc9433dbb4a4ade10609d2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42791 } }
I20260925 16:37:05.389985 21059 leader_election.cc:302] T 00000000000000000000000000000000 P 611a7b99fbc9433dbb4a4ade10609d2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 611a7b99fbc9433dbb4a4ade10609d2d; no voters:
I20260925 16:37:05.390343 21059 leader_election.cc:288] T 00000000000000000000000000000000 P 611a7b99fbc9433dbb4a4ade10609d2d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:37:05.390457 21064 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 611a7b99fbc9433dbb4a4ade10609d2d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:37:05.390723 21064 raft_consensus.cc:686] T 00000000000000000000000000000000 P 611a7b99fbc9433dbb4a4ade10609d2d [term 1 LEADER]: Becoming Leader. State: Replica: 611a7b99fbc9433dbb4a4ade10609d2d, State: Running, Role: LEADER
I20260925 16:37:05.391111 21064 consensus_queue.cc:237] T 00000000000000000000000000000000 P 611a7b99fbc9433dbb4a4ade10609d2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "611a7b99fbc9433dbb4a4ade10609d2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42791 } }
I20260925 16:37:05.391687 21059 sys_catalog.cc:564] T 00000000000000000000000000000000 P 611a7b99fbc9433dbb4a4ade10609d2d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:37:05.392340 21064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 611a7b99fbc9433dbb4a4ade10609d2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 611a7b99fbc9433dbb4a4ade10609d2d. Latest consensus state: current_term: 1 leader_uuid: "611a7b99fbc9433dbb4a4ade10609d2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "611a7b99fbc9433dbb4a4ade10609d2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42791 } } }
I20260925 16:37:05.392441 21064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 611a7b99fbc9433dbb4a4ade10609d2d [sys.catalog]: This master's current role is: LEADER
I20260925 16:37:05.392894 21070 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:37:05.393069 21065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 611a7b99fbc9433dbb4a4ade10609d2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "611a7b99fbc9433dbb4a4ade10609d2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "611a7b99fbc9433dbb4a4ade10609d2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42791 } } }
I20260925 16:37:05.393230 21065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 611a7b99fbc9433dbb4a4ade10609d2d [sys.catalog]: This master's current role is: LEADER
I20260925 16:37:05.395433 21062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:05.381025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44747 (local address 127.0.0.1:42791)
0925 16:37:05.381506 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:05.381520 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:05.381555 (+ 35us) server_negotiation.cc:407] Connection header received
0925 16:37:05.382380 (+ 825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:05.382403 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:05.382842 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:05.383201 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:05.383900 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:05.385395 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:05.386244 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:05.387038 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:05.387222 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:05.389879 (+ 2657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:37:05.389884 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:37:05.389897 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:37:05.389924 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:37:05.392006 (+ 2082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:05.392513 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:05.392522 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:05.392530 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:05.392671 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:05.393027 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:05.393031 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:05.393033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:05.393527 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:37:05.393724 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:05.394178 (+ 454us) server_negotiation.cc:299] Negotiation successful
0925 16:37:05.394471 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":142,"threads_started":1}
I20260925 16:37:05.396114 21070 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:37:05.398645 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:37:05.401979 21070 catalog_manager.cc:1347] Generated new cluster ID: 444bfd03b51e4035b5020bcd6fd3c677
I20260925 16:37:05.402045 21070 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:37:05.410656 21062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:05.400842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41709 (local address 127.0.0.1:42791)
0925 16:37:05.401023 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:05.401028 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:05.402341 (+ 1313us) server_negotiation.cc:407] Connection header received
0925 16:37:05.402477 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:05.402480 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:05.402533 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:05.402650 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:05.403446 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:05.404235 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:05.405301 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:05.405706 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:05.405812 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:05.407330 (+ 1518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:37:05.407335 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:37:05.407338 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:37:05.407371 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:37:05.409014 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:05.409453 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:05.409457 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:05.409459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:05.409543 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:05.409796 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:05.409799 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:05.409801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:05.410000 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:37:05.410120 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:05.410329 (+ 209us) server_negotiation.cc:299] Negotiation successful
0925 16:37:05.410447 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260925 16:37:05.417637 21070 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:37:05.418532 21070 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:37:05.422765 21070 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 611a7b99fbc9433dbb4a4ade10609d2d: Generated new TSK 0
I20260925 16:37:05.423419 21070 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:37:05.427763 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 20991
2026-09-25T16:37:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1388 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790353966814071-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:37:05 dist-test-slave-65pf krb5kdc[21090](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:37:05 dist-test-slave-65pf krb5kdc[21090](info): set up 2 sockets
Sep 25 16:37:05 dist-test-slave-65pf krb5kdc[21090](info): commencing operation
krb5kdc: starting...
W20260925 16:37:06.458877 392 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790353966814071-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790353966814071-392-0/krb5kdc/test-user.keytab.
Sep 25 16:37:06 dist-test-slave-65pf krb5kdc[21090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:37:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:37:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790353966814071-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790353966814071-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 16:37:06.577986 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790353966814071-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790353966814071-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790353966814071-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38029
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790353966814071-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38029
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:37:06.685083 21103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:37:06.685348 21103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:37:06.685389 21103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:37:06.689297 21103 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:37:06.689360 21103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:37:06.689379 21103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:37:06.689395 21103 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:37:06.689410 21103 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:37:06.694634 21103 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790353966814071-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790353966814071-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38029
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790353966814071-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790353966814071-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:37:06.695621 21103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:37:06.696586 21103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 16:37:06.704823 21103 server_base.cc:1034] running on GCE node
W20260925 16:37:06.704794 21111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:37:06.704802 21108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:37:06.704989 21109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:37:06.705507 21103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:37:06.706856 21103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:37:06.710142 21103 hybrid_clock.cc:648] HybridClock initialized: now 1790354226710111 us; error 61 us; skew 500 ppm
Sep 25 16:37:06 dist-test-slave-65pf krb5kdc[21090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354226, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:37:06.712992 21103 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 16:37:06.713912 21103 webserver.cc:460] Webserver started at http://127.0.0.1:36653/ using document root <none> and password file <none>
I20260925 16:37:06.714548 21103 fs_manager.cc:359] Metadata directory not provided
I20260925 16:37:06.714680 21103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:37:06.714920 21103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:37:06.717041 21103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790353966814071-392-0/minicluster-data/master-0/data/instance:
uuid: "320be36772ef469d937db95051c7cea6"
format_stamp: "Formatted at 2026-09-25 16:37:06 on dist-test-slave-65pf"
server_key: "b5bde60f9fd7ad83d6813755de6d0019"
server_key_iv: "641c069aa78ddc5dc0a8c267bd30e9e5"
server_key_version: "encryptionkey@0"
I20260925 16:37:06.717576 21103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790353966814071-392-0/minicluster-data/master-0/wal/instance:
uuid: "320be36772ef469d937db95051c7cea6"
format_stamp: "Formatted at 2026-09-25 16:37:06 on dist-test-slave-65pf"
server_key: "b5bde60f9fd7ad83d6813755de6d0019"
server_key_iv: "641c069aa78ddc5dc0a8c267bd30e9e5"
server_key_version: "encryptionkey@0"
I20260925 16:37:06.721273 21103 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.002s
I20260925 16:37:06.723714 21118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:37:06.724978 21103 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:37:06.725135 21103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790353966814071-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790353966814071-392-0/minicluster-data/master-0/wal
uuid: "320be36772ef469d937db95051c7cea6"
format_stamp: "Formatted at 2026-09-25 16:37:06 on dist-test-slave-65pf"
server_key: "b5bde60f9fd7ad83d6813755de6d0019"
server_key_iv: "641c069aa78ddc5dc0a8c267bd30e9e5"
server_key_version: "encryptionkey@0"
I20260925 16:37:06.725255 21103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790353966814071-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790353966814071-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790353966814071-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:37:06.768388 21103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:37:06.771625 21103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:37:06.771867 21103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:37:06.779089 21103 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38029
I20260925 16:37:06.779109 21170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38029 every 8 connection(s)
I20260925 16:37:06.780061 21103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:37:06.782801 21171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:37:06.784376 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 21103
I20260925 16:37:06.784533 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790353966814071-392-0/minicluster-data/master-0/wal/instance
I20260925 16:37:06.784803 392 external_mini_cluster.cc:1428] Setting key 9f97cc25b5fd87a9fcab1d7ff4472a33
I20260925 16:37:06.788853 21171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 320be36772ef469d937db95051c7cea6: Bootstrap starting.
I20260925 16:37:06.791095 21171 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 320be36772ef469d937db95051c7cea6: Neither blocks nor log segments found. Creating new log.
I20260925 16:37:06.791975 21171 log.cc:824] T 00000000000000000000000000000000 P 320be36772ef469d937db95051c7cea6: Log is configured to *not* fsync() on all Append() calls
I20260925 16:37:06.793795 21174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:06.786220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41111 (local address 127.0.0.1:38029)
0925 16:37:06.786694 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:06.786709 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:06.786745 (+ 36us) server_negotiation.cc:407] Connection header received
0925 16:37:06.787368 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:06.787383 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:06.787713 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:06.788043 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:06.788444 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:06.789456 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:06.790349 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:06.790912 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:06.791100 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:06.791960 (+ 860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:37:06.791965 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:37:06.791976 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:37:06.792487 (+ 511us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:37:06.792551 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:37:06.792581 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:37:06.792685 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:06.792882 (+ 197us) server_negotiation.cc:299] Negotiation successful
0925 16:37:06.793096 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":124,"threads_started":1}
I20260925 16:37:06.794006 21171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 320be36772ef469d937db95051c7cea6: No bootstrap required, opened a new log
I20260925 16:37:06.796648 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:37:06.798456 21171 raft_consensus.cc:348] T 00000000000000000000000000000000 P 320be36772ef469d937db95051c7cea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "320be36772ef469d937db95051c7cea6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38029 } }
I20260925 16:37:06.798746 21171 raft_consensus.cc:374] T 00000000000000000000000000000000 P 320be36772ef469d937db95051c7cea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:37:06.798884 21171 raft_consensus.cc:729] T 00000000000000000000000000000000 P 320be36772ef469d937db95051c7cea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 320be36772ef469d937db95051c7cea6, State: Initialized, Role: FOLLOWER
I20260925 16:37:06.799394 21171 consensus_queue.cc:260] T 00000000000000000000000000000000 P 320be36772ef469d937db95051c7cea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "320be36772ef469d937db95051c7cea6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38029 } }
I20260925 16:37:06.799562 21171 raft_consensus.cc:388] T 00000000000000000000000000000000 P 320be36772ef469d937db95051c7cea6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:37:06.799629 21171 raft_consensus.cc:482] T 00000000000000000000000000000000 P 320be36772ef469d937db95051c7cea6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:37:06.799731 21171 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 320be36772ef469d937db95051c7cea6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:37:06.800876 21171 raft_consensus.cc:504] T 00000000000000000000000000000000 P 320be36772ef469d937db95051c7cea6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "320be36772ef469d937db95051c7cea6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38029 } }
I20260925 16:37:06.801223 21171 leader_election.cc:302] T 00000000000000000000000000000000 P 320be36772ef469d937db95051c7cea6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 320be36772ef469d937db95051c7cea6; no voters:
I20260925 16:37:06.801553 21171 leader_election.cc:288] T 00000000000000000000000000000000 P 320be36772ef469d937db95051c7cea6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:37:06.801677 21181 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 320be36772ef469d937db95051c7cea6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:37:06.801913 21181 raft_consensus.cc:686] T 00000000000000000000000000000000 P 320be36772ef469d937db95051c7cea6 [term 1 LEADER]: Becoming Leader. State: Replica: 320be36772ef469d937db95051c7cea6, State: Running, Role: LEADER
I20260925 16:37:06.802258 21181 consensus_queue.cc:237] T 00000000000000000000000000000000 P 320be36772ef469d937db95051c7cea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "320be36772ef469d937db95051c7cea6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38029 } }
I20260925 16:37:06.803256 21174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:06.798938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40159 (local address 127.0.0.1:38029)
0925 16:37:06.799053 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:06.799057 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:06.799238 (+ 181us) server_negotiation.cc:407] Connection header received
0925 16:37:06.799316 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:06.799318 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:06.799362 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:06.799444 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:06.799786 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:06.800516 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:06.801340 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:06.801732 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:06.801838 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:06.802779 (+ 941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:37:06.802782 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:37:06.802785 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:37:06.802798 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:37:06.802826 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:37:06.802838 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:37:06.802918 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:06.803077 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 16:37:06.803132 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 16:37:06.803308 21171 sys_catalog.cc:564] T 00000000000000000000000000000000 P 320be36772ef469d937db95051c7cea6 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:37:06.804193 21183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 320be36772ef469d937db95051c7cea6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 320be36772ef469d937db95051c7cea6. Latest consensus state: current_term: 1 leader_uuid: "320be36772ef469d937db95051c7cea6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "320be36772ef469d937db95051c7cea6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38029 } } }
I20260925 16:37:06.804322 21183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 320be36772ef469d937db95051c7cea6 [sys.catalog]: This master's current role is: LEADER
I20260925 16:37:06.804725 21182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 320be36772ef469d937db95051c7cea6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "320be36772ef469d937db95051c7cea6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "320be36772ef469d937db95051c7cea6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38029 } } }
I20260925 16:37:06.804822 21182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 320be36772ef469d937db95051c7cea6 [sys.catalog]: This master's current role is: LEADER
I20260925 16:37:06.805096 21190 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:37:06.807937 21190 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:37:06.813066 21190 catalog_manager.cc:1347] Generated new cluster ID: 6d1b3caa4c594cf299f7ed1399435583
I20260925 16:37:06.813200 21190 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:37:06.825953 21190 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:37:06.827270 21190 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:37:06.834395 21190 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 320be36772ef469d937db95051c7cea6: Generated new TSK 0
I20260925 16:37:06.835186 21190 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:37:06.839774 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 21103
2026-09-25T16:37:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1410 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790353966814071-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:37:06 dist-test-slave-65pf krb5kdc[21202](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:37:06 dist-test-slave-65pf krb5kdc[21202](info): set up 2 sockets
Sep 25 16:37:06 dist-test-slave-65pf krb5kdc[21202](info): commencing operation
krb5kdc: starting...
W20260925 16:37:07.858798 392 mini_kdc.cc:121] Time spent starting KDC: real 1.000s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790353966814071-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790353966814071-392-0/krb5kdc/test-user.keytab.
Sep 25 16:37:07 dist-test-slave-65pf krb5kdc[21202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:37:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:37:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790353966814071-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790353966814071-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 16:37:07.981779 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790353966814071-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790353966814071-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790353966814071-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34151
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790353966814071-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34151
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:37:08.090349 21215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:37:08.090692 21215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:37:08.090807 21215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:37:08.094813 21215 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:37:08.094906 21215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:37:08.094942 21215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:37:08.094978 21215 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:37:08.095006 21215 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:37:08.100857 21215 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790353966814071-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790353966814071-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34151
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790353966814071-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21215
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790353966814071-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:37:08.101908 21215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:37:08.102953 21215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:37:08.111495 21220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:37:08.111512 21221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:37:08.111514 21223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:37:08.111744 21215 server_base.cc:1034] running on GCE node
I20260925 16:37:08.112423 21215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:37:08.113507 21215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:37:08.114698 21215 hybrid_clock.cc:648] HybridClock initialized: now 1790354228114663 us; error 46 us; skew 500 ppm
Sep 25 16:37:08 dist-test-slave-65pf krb5kdc[21202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354228, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:37:08.117547 21215 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 16:37:08.118490 21215 webserver.cc:460] Webserver started at http://127.0.0.1:43487/ using document root <none> and password file <none>
I20260925 16:37:08.119122 21215 fs_manager.cc:359] Metadata directory not provided
I20260925 16:37:08.119226 21215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:37:08.119472 21215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:37:08.121600 21215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790353966814071-392-0/minicluster-data/master-0/data/instance:
uuid: "150711e201f149479393c75e98f3c0bd"
format_stamp: "Formatted at 2026-09-25 16:37:08 on dist-test-slave-65pf"
server_key: "3a872e3921be6a5140ef2ae157931ae3"
server_key_iv: "8822790bb9f9a1bc298543b6ca40588a"
server_key_version: "encryptionkey@0"
I20260925 16:37:08.122195 21215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790353966814071-392-0/minicluster-data/master-0/wal/instance:
uuid: "150711e201f149479393c75e98f3c0bd"
format_stamp: "Formatted at 2026-09-25 16:37:08 on dist-test-slave-65pf"
server_key: "3a872e3921be6a5140ef2ae157931ae3"
server_key_iv: "8822790bb9f9a1bc298543b6ca40588a"
server_key_version: "encryptionkey@0"
I20260925 16:37:08.126148 21215 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 16:37:08.128648 21230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:37:08.129890 21215 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 16:37:08.130064 21215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790353966814071-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790353966814071-392-0/minicluster-data/master-0/wal
uuid: "150711e201f149479393c75e98f3c0bd"
format_stamp: "Formatted at 2026-09-25 16:37:08 on dist-test-slave-65pf"
server_key: "3a872e3921be6a5140ef2ae157931ae3"
server_key_iv: "8822790bb9f9a1bc298543b6ca40588a"
server_key_version: "encryptionkey@0"
I20260925 16:37:08.130182 21215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790353966814071-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790353966814071-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790353966814071-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:37:08.134893 21215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:37:08.138200 21215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:37:08.138396 21215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:37:08.145376 21215 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34151
I20260925 16:37:08.145401 21282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34151 every 8 connection(s)
I20260925 16:37:08.146348 21215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:37:08.147998 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 21215
I20260925 16:37:08.148170 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790353966814071-392-0/minicluster-data/master-0/wal/instance
I20260925 16:37:08.148463 392 external_mini_cluster.cc:1428] Setting key 10ad04130b94407b6ac500cb7db930c9
I20260925 16:37:08.149500 21283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:37:08.153826 21286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:08.149922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35393 (local address 127.0.0.1:34151)
0925 16:37:08.150382 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:08.150397 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:08.150433 (+ 36us) server_negotiation.cc:407] Connection header received
0925 16:37:08.151137 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:08.151154 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:08.151450 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:08.151802 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:08.152633 (+ 831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:37:08.152637 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:37:08.152649 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:37:08.152659 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:37:08.152713 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:37:08.152734 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:37:08.152888 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:08.152960 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 16:37:08.153121 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":139,"threads_started":1}
I20260925 16:37:08.155876 21283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 150711e201f149479393c75e98f3c0bd: Bootstrap starting.
I20260925 16:37:08.156942 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:37:08.158459 21283 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 150711e201f149479393c75e98f3c0bd: Neither blocks nor log segments found. Creating new log.
I20260925 16:37:08.159360 21283 log.cc:824] T 00000000000000000000000000000000 P 150711e201f149479393c75e98f3c0bd: Log is configured to *not* fsync() on all Append() calls
I20260925 16:37:08.160929 21286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:08.159052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43329 (local address 127.0.0.1:34151)
0925 16:37:08.159173 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:08.159177 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:08.159369 (+ 192us) server_negotiation.cc:407] Connection header received
0925 16:37:08.159515 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:08.159518 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:08.159567 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:08.159635 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:08.160507 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:37:08.160510 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:37:08.160512 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:37:08.160523 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:37:08.160549 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:37:08.160559 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:37:08.160633 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:08.160742 (+ 109us) server_negotiation.cc:299] Negotiation successful
0925 16:37:08.160776 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 16:37:08.161772 21283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 150711e201f149479393c75e98f3c0bd: No bootstrap required, opened a new log
I20260925 16:37:08.165282 21283 raft_consensus.cc:348] T 00000000000000000000000000000000 P 150711e201f149479393c75e98f3c0bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "150711e201f149479393c75e98f3c0bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34151 } }
I20260925 16:37:08.165529 21283 raft_consensus.cc:374] T 00000000000000000000000000000000 P 150711e201f149479393c75e98f3c0bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:37:08.165565 21283 raft_consensus.cc:729] T 00000000000000000000000000000000 P 150711e201f149479393c75e98f3c0bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 150711e201f149479393c75e98f3c0bd, State: Initialized, Role: FOLLOWER
I20260925 16:37:08.165983 21283 consensus_queue.cc:260] T 00000000000000000000000000000000 P 150711e201f149479393c75e98f3c0bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "150711e201f149479393c75e98f3c0bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34151 } }
I20260925 16:37:08.166108 21283 raft_consensus.cc:388] T 00000000000000000000000000000000 P 150711e201f149479393c75e98f3c0bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:37:08.166191 21283 raft_consensus.cc:482] T 00000000000000000000000000000000 P 150711e201f149479393c75e98f3c0bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:37:08.166309 21283 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 150711e201f149479393c75e98f3c0bd [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:37:08.167294 21283 raft_consensus.cc:504] T 00000000000000000000000000000000 P 150711e201f149479393c75e98f3c0bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "150711e201f149479393c75e98f3c0bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34151 } }
I20260925 16:37:08.167656 21283 leader_election.cc:302] T 00000000000000000000000000000000 P 150711e201f149479393c75e98f3c0bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 150711e201f149479393c75e98f3c0bd; no voters:
I20260925 16:37:08.168015 21283 leader_election.cc:288] T 00000000000000000000000000000000 P 150711e201f149479393c75e98f3c0bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:37:08.168136 21293 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 150711e201f149479393c75e98f3c0bd [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:37:08.168435 21293 raft_consensus.cc:686] T 00000000000000000000000000000000 P 150711e201f149479393c75e98f3c0bd [term 1 LEADER]: Becoming Leader. State: Replica: 150711e201f149479393c75e98f3c0bd, State: Running, Role: LEADER
I20260925 16:37:08.168831 21293 consensus_queue.cc:237] T 00000000000000000000000000000000 P 150711e201f149479393c75e98f3c0bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "150711e201f149479393c75e98f3c0bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34151 } }
I20260925 16:37:08.169260 21283 sys_catalog.cc:564] T 00000000000000000000000000000000 P 150711e201f149479393c75e98f3c0bd [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:37:08.170701 21295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 150711e201f149479393c75e98f3c0bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 150711e201f149479393c75e98f3c0bd. Latest consensus state: current_term: 1 leader_uuid: "150711e201f149479393c75e98f3c0bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "150711e201f149479393c75e98f3c0bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34151 } } }
I20260925 16:37:08.170717 21294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 150711e201f149479393c75e98f3c0bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "150711e201f149479393c75e98f3c0bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "150711e201f149479393c75e98f3c0bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34151 } } }
I20260925 16:37:08.170862 21295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 150711e201f149479393c75e98f3c0bd [sys.catalog]: This master's current role is: LEADER
I20260925 16:37:08.170862 21294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 150711e201f149479393c75e98f3c0bd [sys.catalog]: This master's current role is: LEADER
W20260925 16:37:08.173689 21308 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 150711e201f149479393c75e98f3c0bd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 16:37:08.173782 21308 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 16:37:08.173870 21307 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:37:08.174569 21307 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:37:08.180838 21307 catalog_manager.cc:1347] Generated new cluster ID: 59d413ada4584c108d8e3b71ac7fe94e
I20260925 16:37:08.180924 21307 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:37:08.192911 21307 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:37:08.193244 21307 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:37:08.205499 21307 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 150711e201f149479393c75e98f3c0bd: Generated new TSK 0
I20260925 16:37:08.206421 21307 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:37:08.219940 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 21215
2026-09-25T16:37:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1380 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790353966814071-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:37:08 dist-test-slave-65pf krb5kdc[21314](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:37:08 dist-test-slave-65pf krb5kdc[21314](info): set up 2 sockets
Sep 25 16:37:08 dist-test-slave-65pf krb5kdc[21314](info): commencing operation
krb5kdc: starting...
W20260925 16:37:09.258780 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790353966814071-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790353966814071-392-0/krb5kdc/test-user.keytab.
Sep 25 16:37:09 dist-test-slave-65pf krb5kdc[21314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:37:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:37:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790353966814071-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790353966814071-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 16:37:09.377828 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790353966814071-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790353966814071-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790353966814071-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34415
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790353966814071-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34415
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:37:09.485699 21327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:37:09.485952 21327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:37:09.485994 21327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:37:09.490020 21327 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:37:09.490082 21327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:37:09.490100 21327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:37:09.490116 21327 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:37:09.490131 21327 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:37:09.495357 21327 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790353966814071-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790353966814071-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34415
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790353966814071-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790353966814071-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:37:09.496309 21327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:37:09.497226 21327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:37:09.505338 21332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:37:09.505544 21327 server_base.cc:1034] running on GCE node
W20260925 16:37:09.505337 21335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:37:09.505338 21333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:37:09.506177 21327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:37:09.507316 21327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:37:09.508523 21327 hybrid_clock.cc:648] HybridClock initialized: now 1790354229508493 us; error 40 us; skew 500 ppm
Sep 25 16:37:09 dist-test-slave-65pf krb5kdc[21314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354229, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:37:09.511297 21327 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 16:37:09.512174 21327 webserver.cc:460] Webserver started at http://127.0.0.1:45467/ using document root <none> and password file <none>
I20260925 16:37:09.512734 21327 fs_manager.cc:359] Metadata directory not provided
I20260925 16:37:09.512827 21327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:37:09.513078 21327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:37:09.515198 21327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790353966814071-392-0/minicluster-data/master-0/data/instance:
uuid: "fa6c3abc96eb41ab9699b082099e5cc6"
format_stamp: "Formatted at 2026-09-25 16:37:09 on dist-test-slave-65pf"
server_key: "4593352e0bff836fe495e5298aa9b69f"
server_key_iv: "6bbd24928990a7a34c4d1692f39f72cd"
server_key_version: "encryptionkey@0"
I20260925 16:37:09.515700 21327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790353966814071-392-0/minicluster-data/master-0/wal/instance:
uuid: "fa6c3abc96eb41ab9699b082099e5cc6"
format_stamp: "Formatted at 2026-09-25 16:37:09 on dist-test-slave-65pf"
server_key: "4593352e0bff836fe495e5298aa9b69f"
server_key_iv: "6bbd24928990a7a34c4d1692f39f72cd"
server_key_version: "encryptionkey@0"
I20260925 16:37:09.519284 21327 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 16:37:09.521786 21342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:37:09.523058 21327 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 16:37:09.523155 21327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790353966814071-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790353966814071-392-0/minicluster-data/master-0/wal
uuid: "fa6c3abc96eb41ab9699b082099e5cc6"
format_stamp: "Formatted at 2026-09-25 16:37:09 on dist-test-slave-65pf"
server_key: "4593352e0bff836fe495e5298aa9b69f"
server_key_iv: "6bbd24928990a7a34c4d1692f39f72cd"
server_key_version: "encryptionkey@0"
I20260925 16:37:09.523262 21327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790353966814071-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790353966814071-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790353966814071-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:37:09.547396 21327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:37:09.550287 21327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:37:09.550442 21327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:37:09.557269 21327 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34415
I20260925 16:37:09.557286 21394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34415 every 8 connection(s)
I20260925 16:37:09.558219 21327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:37:09.560978 21395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:37:09.564095 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 21327
I20260925 16:37:09.564250 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790353966814071-392-0/minicluster-data/master-0/wal/instance
I20260925 16:37:09.564538 392 external_mini_cluster.cc:1428] Setting key 6fb91f0421d5a945cebfcf03a0839cb5
I20260925 16:37:09.567065 21395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa6c3abc96eb41ab9699b082099e5cc6: Bootstrap starting.
I20260925 16:37:09.569147 21395 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa6c3abc96eb41ab9699b082099e5cc6: Neither blocks nor log segments found. Creating new log.
I20260925 16:37:09.569887 21395 log.cc:824] T 00000000000000000000000000000000 P fa6c3abc96eb41ab9699b082099e5cc6: Log is configured to *not* fsync() on all Append() calls
I20260925 16:37:09.571954 21395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa6c3abc96eb41ab9699b082099e5cc6: No bootstrap required, opened a new log
Sep 25 16:37:09 dist-test-slave-65pf krb5kdc[21314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 16:37:09.575737 21395 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa6c3abc96eb41ab9699b082099e5cc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa6c3abc96eb41ab9699b082099e5cc6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34415 } }
I20260925 16:37:09.576012 21395 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa6c3abc96eb41ab9699b082099e5cc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:37:09.576115 21395 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa6c3abc96eb41ab9699b082099e5cc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6c3abc96eb41ab9699b082099e5cc6, State: Initialized, Role: FOLLOWER
I20260925 16:37:09.576640 21395 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa6c3abc96eb41ab9699b082099e5cc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa6c3abc96eb41ab9699b082099e5cc6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34415 } }
I20260925 16:37:09.576815 21395 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa6c3abc96eb41ab9699b082099e5cc6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:37:09.576897 21395 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa6c3abc96eb41ab9699b082099e5cc6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:37:09.577011 21395 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa6c3abc96eb41ab9699b082099e5cc6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:37:09.578132 21395 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa6c3abc96eb41ab9699b082099e5cc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa6c3abc96eb41ab9699b082099e5cc6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34415 } }
I20260925 16:37:09.578548 21395 leader_election.cc:302] T 00000000000000000000000000000000 P fa6c3abc96eb41ab9699b082099e5cc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa6c3abc96eb41ab9699b082099e5cc6; no voters:
I20260925 16:37:09.578948 21395 leader_election.cc:288] T 00000000000000000000000000000000 P fa6c3abc96eb41ab9699b082099e5cc6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:37:09.579061 21400 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa6c3abc96eb41ab9699b082099e5cc6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:37:09.579288 21400 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa6c3abc96eb41ab9699b082099e5cc6 [term 1 LEADER]: Becoming Leader. State: Replica: fa6c3abc96eb41ab9699b082099e5cc6, State: Running, Role: LEADER
I20260925 16:37:09.579641 21400 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa6c3abc96eb41ab9699b082099e5cc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa6c3abc96eb41ab9699b082099e5cc6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34415 } }
I20260925 16:37:09.580041 21398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:09.565977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.106:45255 (local address 127.0.0.1:34415)
0925 16:37:09.566400 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:09.566414 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:09.566449 (+ 35us) server_negotiation.cc:407] Connection header received
0925 16:37:09.567103 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:09.567142 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:09.567508 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:09.567836 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:09.568626 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:09.569689 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:09.570793 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:09.571261 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:09.571443 (+ 182us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:09.574214 (+ 2771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:37:09.574219 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:37:09.574231 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:37:09.574257 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:37:09.576640 (+ 2383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:09.577239 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:09.577243 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:09.577248 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:09.577362 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:09.577736 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:09.577738 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:09.577739 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:09.578098 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:37:09.578249 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:09.578816 (+ 567us) server_negotiation.cc:299] Negotiation successful
0925 16:37:09.579051 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":120,"threads_started":1}
I20260925 16:37:09.580314 21395 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa6c3abc96eb41ab9699b082099e5cc6 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:37:09.580826 21400 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa6c3abc96eb41ab9699b082099e5cc6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa6c3abc96eb41ab9699b082099e5cc6. Latest consensus state: current_term: 1 leader_uuid: "fa6c3abc96eb41ab9699b082099e5cc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa6c3abc96eb41ab9699b082099e5cc6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34415 } } }
I20260925 16:37:09.581410 21401 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa6c3abc96eb41ab9699b082099e5cc6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa6c3abc96eb41ab9699b082099e5cc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa6c3abc96eb41ab9699b082099e5cc6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34415 } } }
I20260925 16:37:09.581528 21400 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa6c3abc96eb41ab9699b082099e5cc6 [sys.catalog]: This master's current role is: LEADER
I20260925 16:37:09.581563 21401 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa6c3abc96eb41ab9699b082099e5cc6 [sys.catalog]: This master's current role is: LEADER
I20260925 16:37:09.581975 21404 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:37:09.583485 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:37:09.585068 21404 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:37:09.590525 21404 catalog_manager.cc:1347] Generated new cluster ID: 50337308da6c45328170ec53f416afef
I20260925 16:37:09.590629 21404 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:37:09.594972 21398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:09.586090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.106:45351 (local address 127.0.0.1:34415)
0925 16:37:09.586227 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:09.586232 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:09.586608 (+ 376us) server_negotiation.cc:407] Connection header received
0925 16:37:09.586835 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:09.586839 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:09.586891 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:09.586991 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:09.587711 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:09.588242 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:09.589274 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:09.589537 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:09.589664 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:09.591349 (+ 1685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:37:09.591353 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:37:09.591356 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:37:09.591391 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:37:09.592975 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:09.593532 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:09.593536 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:09.593538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:09.593628 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:09.593999 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:09.594003 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:09.594005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:09.594189 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:37:09.594353 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:09.594687 (+ 334us) server_negotiation.cc:299] Negotiation successful
0925 16:37:09.594804 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 16:37:09.597939 21404 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:37:09.598886 21404 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:37:09.602975 21404 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa6c3abc96eb41ab9699b082099e5cc6: Generated new TSK 0
I20260925 16:37:09.603636 21404 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:37:09.608278 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 21327
W20260925 16:37:09.614487 21312 connection.cc:535] client connection to 127.0.0.1:34415 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34415): Connection reset by peer (error 104)
2026-09-25T16:37:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1389 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790353966814071-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:37:09 dist-test-slave-65pf krb5kdc[21426](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:37:09 dist-test-slave-65pf krb5kdc[21426](info): set up 2 sockets
Sep 25 16:37:09 dist-test-slave-65pf krb5kdc[21426](info): commencing operation
krb5kdc: starting...
W20260925 16:37:10.642822 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790353966814071-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790353966814071-392-0/krb5kdc/test-user.keytab.
Sep 25 16:37:10 dist-test-slave-65pf krb5kdc[21426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:37:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:37:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790353966814071-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790353966814071-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 16:37:10.762979 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790353966814071-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790353966814071-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790353966814071-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34573
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790353966814071-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34573
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:37:10.870905 21439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:37:10.871176 21439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:37:10.871220 21439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:37:10.875317 21439 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:37:10.875381 21439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:37:10.875401 21439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:37:10.875417 21439 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:37:10.875470 21439 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:37:10.880748 21439 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790353966814071-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790353966814071-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34573
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790353966814071-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21439
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790353966814071-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:37:10.881798 21439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:37:10.882882 21439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:37:10.891494 21445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:37:10.891495 21444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:37:10.891465 21447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:37:10.891844 21439 server_base.cc:1034] running on GCE node
I20260925 16:37:10.892463 21439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:37:10.893601 21439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:37:10.894783 21439 hybrid_clock.cc:648] HybridClock initialized: now 1790354230894759 us; error 38 us; skew 500 ppm
Sep 25 16:37:10 dist-test-slave-65pf krb5kdc[21426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354230, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:37:10.904492 21439 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 16:37:10.905542 21439 webserver.cc:460] Webserver started at http://127.0.0.1:42975/ using document root <none> and password file <none>
I20260925 16:37:10.906131 21439 fs_manager.cc:359] Metadata directory not provided
I20260925 16:37:10.906231 21439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:37:10.906471 21439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:37:10.908639 21439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790353966814071-392-0/minicluster-data/master-0/data/instance:
uuid: "ebc48539a4df4ad0a2d6036422652ff1"
format_stamp: "Formatted at 2026-09-25 16:37:10 on dist-test-slave-65pf"
server_key: "a0ba2b2d56204a8c7ccaeb5ee13284fc"
server_key_iv: "e5cd9ac4d0fc04881316af5fd614396f"
server_key_version: "encryptionkey@0"
I20260925 16:37:10.909231 21439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790353966814071-392-0/minicluster-data/master-0/wal/instance:
uuid: "ebc48539a4df4ad0a2d6036422652ff1"
format_stamp: "Formatted at 2026-09-25 16:37:10 on dist-test-slave-65pf"
server_key: "a0ba2b2d56204a8c7ccaeb5ee13284fc"
server_key_iv: "e5cd9ac4d0fc04881316af5fd614396f"
server_key_version: "encryptionkey@0"
I20260925 16:37:10.913062 21439 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 16:37:10.915452 21454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:37:10.916610 21439 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:37:10.916780 21439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790353966814071-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790353966814071-392-0/minicluster-data/master-0/wal
uuid: "ebc48539a4df4ad0a2d6036422652ff1"
format_stamp: "Formatted at 2026-09-25 16:37:10 on dist-test-slave-65pf"
server_key: "a0ba2b2d56204a8c7ccaeb5ee13284fc"
server_key_iv: "e5cd9ac4d0fc04881316af5fd614396f"
server_key_version: "encryptionkey@0"
I20260925 16:37:10.916891 21439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790353966814071-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790353966814071-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790353966814071-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:37:10.928208 21439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:37:10.931911 21439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:37:10.932116 21439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:37:10.938851 21439 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34573
I20260925 16:37:10.939136 21506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34573 every 8 connection(s)
I20260925 16:37:10.939815 21439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:37:10.942399 21507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:37:10.947736 21507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebc48539a4df4ad0a2d6036422652ff1: Bootstrap starting.
I20260925 16:37:10.949143 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 21439
I20260925 16:37:10.949285 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790353966814071-392-0/minicluster-data/master-0/wal/instance
I20260925 16:37:10.949586 392 external_mini_cluster.cc:1428] Setting key 8a9001077c0a60a656e0c174cb18aed6
I20260925 16:37:10.950457 21507 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ebc48539a4df4ad0a2d6036422652ff1: Neither blocks nor log segments found. Creating new log.
I20260925 16:37:10.951316 21507 log.cc:824] T 00000000000000000000000000000000 P ebc48539a4df4ad0a2d6036422652ff1: Log is configured to *not* fsync() on all Append() calls
I20260925 16:37:10.953069 21507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebc48539a4df4ad0a2d6036422652ff1: No bootstrap required, opened a new log
I20260925 16:37:10.956280 21507 raft_consensus.cc:348] T 00000000000000000000000000000000 P ebc48539a4df4ad0a2d6036422652ff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebc48539a4df4ad0a2d6036422652ff1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34573 } }
I20260925 16:37:10.956507 21507 raft_consensus.cc:374] T 00000000000000000000000000000000 P ebc48539a4df4ad0a2d6036422652ff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:37:10.956581 21507 raft_consensus.cc:729] T 00000000000000000000000000000000 P ebc48539a4df4ad0a2d6036422652ff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebc48539a4df4ad0a2d6036422652ff1, State: Initialized, Role: FOLLOWER
I20260925 16:37:10.956972 21507 consensus_queue.cc:260] T 00000000000000000000000000000000 P ebc48539a4df4ad0a2d6036422652ff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebc48539a4df4ad0a2d6036422652ff1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34573 } }
I20260925 16:37:10.957127 21507 raft_consensus.cc:388] T 00000000000000000000000000000000 P ebc48539a4df4ad0a2d6036422652ff1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:37:10.957206 21507 raft_consensus.cc:482] T 00000000000000000000000000000000 P ebc48539a4df4ad0a2d6036422652ff1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:37:10.957324 21507 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ebc48539a4df4ad0a2d6036422652ff1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:37:10.958253 21507 raft_consensus.cc:504] T 00000000000000000000000000000000 P ebc48539a4df4ad0a2d6036422652ff1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebc48539a4df4ad0a2d6036422652ff1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34573 } }
I20260925 16:37:10.958613 21507 leader_election.cc:302] T 00000000000000000000000000000000 P ebc48539a4df4ad0a2d6036422652ff1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ebc48539a4df4ad0a2d6036422652ff1; no voters:
I20260925 16:37:10.958659 21510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:10.950754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.106:45641 (local address 127.0.0.1:34573)
0925 16:37:10.951222 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:10.951238 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:10.951275 (+ 37us) server_negotiation.cc:407] Connection header received
0925 16:37:10.951985 (+ 710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:10.952003 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:10.952348 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:10.952692 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:10.953101 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:10.954086 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:10.955085 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:10.955542 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:10.955720 (+ 178us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:10.956703 (+ 983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:37:10.956710 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:37:10.956733 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:37:10.957245 (+ 512us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:37:10.957334 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:37:10.957365 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:37:10.957524 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:10.957715 (+ 191us) server_negotiation.cc:299] Negotiation successful
0925 16:37:10.957843 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":127,"threads_started":1}
I20260925 16:37:10.958976 21507 leader_election.cc:288] T 00000000000000000000000000000000 P ebc48539a4df4ad0a2d6036422652ff1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:37:10.959081 21512 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ebc48539a4df4ad0a2d6036422652ff1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:37:10.959290 21512 raft_consensus.cc:686] T 00000000000000000000000000000000 P ebc48539a4df4ad0a2d6036422652ff1 [term 1 LEADER]: Becoming Leader. State: Replica: ebc48539a4df4ad0a2d6036422652ff1, State: Running, Role: LEADER
I20260925 16:37:10.959573 21512 consensus_queue.cc:237] T 00000000000000000000000000000000 P ebc48539a4df4ad0a2d6036422652ff1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebc48539a4df4ad0a2d6036422652ff1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34573 } }
I20260925 16:37:10.960161 21507 sys_catalog.cc:564] T 00000000000000000000000000000000 P ebc48539a4df4ad0a2d6036422652ff1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:37:10.961073 21513 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebc48539a4df4ad0a2d6036422652ff1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ebc48539a4df4ad0a2d6036422652ff1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebc48539a4df4ad0a2d6036422652ff1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34573 } } }
I20260925 16:37:10.961221 21513 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebc48539a4df4ad0a2d6036422652ff1 [sys.catalog]: This master's current role is: LEADER
I20260925 16:37:10.961606 21521 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:37:10.961571 21514 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebc48539a4df4ad0a2d6036422652ff1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebc48539a4df4ad0a2d6036422652ff1. Latest consensus state: current_term: 1 leader_uuid: "ebc48539a4df4ad0a2d6036422652ff1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebc48539a4df4ad0a2d6036422652ff1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34573 } } }
I20260925 16:37:10.961659 21514 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebc48539a4df4ad0a2d6036422652ff1 [sys.catalog]: This master's current role is: LEADER
I20260925 16:37:10.962066 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:37:10.964701 21521 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:37:10.969401 21510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:10.964221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.106:48577 (local address 127.0.0.1:34573)
0925 16:37:10.964331 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:10.964336 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:10.964717 (+ 381us) server_negotiation.cc:407] Connection header received
0925 16:37:10.964881 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:10.964883 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:10.964933 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:10.964994 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:10.965479 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:10.966013 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:10.966956 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:10.967265 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:10.967335 (+ 70us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:10.968448 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:37:10.968452 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:37:10.968455 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:37:10.968469 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:37:10.968499 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:37:10.968515 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:37:10.968615 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:10.969223 (+ 608us) server_negotiation.cc:299] Negotiation successful
0925 16:37:10.969262 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 16:37:10.970706 21521 catalog_manager.cc:1347] Generated new cluster ID: a2d16aa494434293bd9965e32c723e21
I20260925 16:37:10.970764 21521 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:37:10.980120 21521 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:37:10.981123 21521 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:37:10.984548 21521 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ebc48539a4df4ad0a2d6036422652ff1: Generated new TSK 0
I20260925 16:37:10.985266 21521 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:37:10.991465 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 21439
W20260925 16:37:10.996865 21424 connection.cc:535] client connection to 127.0.0.1:34573 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34573): Connection reset by peer (error 104)
2026-09-25T16:37:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1382 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790353966814071-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:37:11 dist-test-slave-65pf krb5kdc[21538](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:37:11 dist-test-slave-65pf krb5kdc[21538](info): set up 2 sockets
Sep 25 16:37:11 dist-test-slave-65pf krb5kdc[21538](info): commencing operation
krb5kdc: starting...
W20260925 16:37:12.015874 392 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790353966814071-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790353966814071-392-0/krb5kdc/test-user.keytab.
Sep 25 16:37:12 dist-test-slave-65pf krb5kdc[21538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:37:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:37:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790353966814071-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790353966814071-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 16:37:12.146402 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790353966814071-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790353966814071-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790353966814071-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45217
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790353966814071-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45217
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:37:12.256003 21551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:37:12.256327 21551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:37:12.256400 21551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:37:12.260438 21551 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:37:12.260545 21551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:37:12.260586 21551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:37:12.260648 21551 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:37:12.260687 21551 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:37:12.265974 21551 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790353966814071-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790353966814071-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45217
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790353966814071-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21551
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790353966814071-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:37:12.267083 21551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:37:12.268103 21551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:37:12.276369 21556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:37:12.276348 21559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:37:12.276517 21551 server_base.cc:1034] running on GCE node
W20260925 16:37:12.276796 21557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:37:12.277351 21551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:37:12.278465 21551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:37:12.279687 21551 hybrid_clock.cc:648] HybridClock initialized: now 1790354232279650 us; error 44 us; skew 500 ppm
Sep 25 16:37:12 dist-test-slave-65pf krb5kdc[21538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354232, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:37:12.282716 21551 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 16:37:12.283708 21551 webserver.cc:460] Webserver started at http://127.0.0.1:46161/ using document root <none> and password file <none>
I20260925 16:37:12.284299 21551 fs_manager.cc:359] Metadata directory not provided
I20260925 16:37:12.284399 21551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:37:12.284648 21551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:37:12.286917 21551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790353966814071-392-0/minicluster-data/master-0/data/instance:
uuid: "4432fa4c6a9e41d4a2332ad8abb8f61f"
format_stamp: "Formatted at 2026-09-25 16:37:12 on dist-test-slave-65pf"
server_key: "310bebd9459e6a67f35fd4f371e64c18"
server_key_iv: "0d2f27fc460016d192a7a40d04445216"
server_key_version: "encryptionkey@0"
I20260925 16:37:12.287521 21551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790353966814071-392-0/minicluster-data/master-0/wal/instance:
uuid: "4432fa4c6a9e41d4a2332ad8abb8f61f"
format_stamp: "Formatted at 2026-09-25 16:37:12 on dist-test-slave-65pf"
server_key: "310bebd9459e6a67f35fd4f371e64c18"
server_key_iv: "0d2f27fc460016d192a7a40d04445216"
server_key_version: "encryptionkey@0"
I20260925 16:37:12.291446 21551 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 16:37:12.294075 21566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:37:12.295382 21551 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:37:12.295531 21551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790353966814071-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790353966814071-392-0/minicluster-data/master-0/wal
uuid: "4432fa4c6a9e41d4a2332ad8abb8f61f"
format_stamp: "Formatted at 2026-09-25 16:37:12 on dist-test-slave-65pf"
server_key: "310bebd9459e6a67f35fd4f371e64c18"
server_key_iv: "0d2f27fc460016d192a7a40d04445216"
server_key_version: "encryptionkey@0"
I20260925 16:37:12.295646 21551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790353966814071-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790353966814071-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790353966814071-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:37:12.300267 21551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:37:12.303475 21551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:37:12.303670 21551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:37:12.310465 21551 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45217
I20260925 16:37:12.310477 21618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45217 every 8 connection(s)
I20260925 16:37:12.311473 21551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:37:12.312495 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 21551
I20260925 16:37:12.312644 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790353966814071-392-0/minicluster-data/master-0/wal/instance
I20260925 16:37:12.312947 392 external_mini_cluster.cc:1428] Setting key 1b21c1f36fb4404dd975fed95bcc6632
I20260925 16:37:12.314765 21619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:37:12.317960 21622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:12.314254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.106:35951 (local address 127.0.0.1:45217)
0925 16:37:12.314670 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:12.314686 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:12.314722 (+ 36us) server_negotiation.cc:407] Connection header received
0925 16:37:12.315387 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:12.315403 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:12.315676 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:12.315953 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:12.316789 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:37:12.316793 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:37:12.316804 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:37:12.316814 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:37:12.316865 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:37:12.316883 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:37:12.316989 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:12.317105 (+ 116us) server_negotiation.cc:299] Negotiation successful
0925 16:37:12.317286 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":133,"threads_started":1}
I20260925 16:37:12.320926 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:37:12.321017 21619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4432fa4c6a9e41d4a2332ad8abb8f61f: Bootstrap starting.
I20260925 16:37:12.323750 21619 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4432fa4c6a9e41d4a2332ad8abb8f61f: Neither blocks nor log segments found. Creating new log.
I20260925 16:37:12.324532 21619 log.cc:824] T 00000000000000000000000000000000 P 4432fa4c6a9e41d4a2332ad8abb8f61f: Log is configured to *not* fsync() on all Append() calls
I20260925 16:37:12.324824 21622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:12.323146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.106:53391 (local address 127.0.0.1:45217)
0925 16:37:12.323267 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:12.323271 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:12.323466 (+ 195us) server_negotiation.cc:407] Connection header received
0925 16:37:12.323547 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:12.323550 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:12.323589 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:12.323681 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:12.324446 (+ 765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:37:12.324449 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:37:12.324452 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:37:12.324463 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:37:12.324490 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:37:12.324502 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:37:12.324599 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:12.324673 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 16:37:12.324710 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 16:37:12.326489 21619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4432fa4c6a9e41d4a2332ad8abb8f61f: No bootstrap required, opened a new log
I20260925 16:37:12.329888 21619 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4432fa4c6a9e41d4a2332ad8abb8f61f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4432fa4c6a9e41d4a2332ad8abb8f61f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45217 } }
I20260925 16:37:12.330106 21619 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4432fa4c6a9e41d4a2332ad8abb8f61f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:37:12.330147 21619 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4432fa4c6a9e41d4a2332ad8abb8f61f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4432fa4c6a9e41d4a2332ad8abb8f61f, State: Initialized, Role: FOLLOWER
I20260925 16:37:12.330498 21619 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4432fa4c6a9e41d4a2332ad8abb8f61f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4432fa4c6a9e41d4a2332ad8abb8f61f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45217 } }
I20260925 16:37:12.330662 21619 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4432fa4c6a9e41d4a2332ad8abb8f61f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:37:12.330718 21619 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4432fa4c6a9e41d4a2332ad8abb8f61f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:37:12.330791 21619 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4432fa4c6a9e41d4a2332ad8abb8f61f [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:37:12.331712 21619 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4432fa4c6a9e41d4a2332ad8abb8f61f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4432fa4c6a9e41d4a2332ad8abb8f61f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45217 } }
I20260925 16:37:12.332047 21619 leader_election.cc:302] T 00000000000000000000000000000000 P 4432fa4c6a9e41d4a2332ad8abb8f61f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4432fa4c6a9e41d4a2332ad8abb8f61f; no voters:
I20260925 16:37:12.332361 21619 leader_election.cc:288] T 00000000000000000000000000000000 P 4432fa4c6a9e41d4a2332ad8abb8f61f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:37:12.332469 21629 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4432fa4c6a9e41d4a2332ad8abb8f61f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:37:12.332695 21629 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4432fa4c6a9e41d4a2332ad8abb8f61f [term 1 LEADER]: Becoming Leader. State: Replica: 4432fa4c6a9e41d4a2332ad8abb8f61f, State: Running, Role: LEADER
I20260925 16:37:12.333026 21629 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4432fa4c6a9e41d4a2332ad8abb8f61f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4432fa4c6a9e41d4a2332ad8abb8f61f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45217 } }
I20260925 16:37:12.333590 21619 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4432fa4c6a9e41d4a2332ad8abb8f61f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:37:12.334648 21630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4432fa4c6a9e41d4a2332ad8abb8f61f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4432fa4c6a9e41d4a2332ad8abb8f61f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4432fa4c6a9e41d4a2332ad8abb8f61f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45217 } } }
I20260925 16:37:12.334775 21630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4432fa4c6a9e41d4a2332ad8abb8f61f [sys.catalog]: This master's current role is: LEADER
I20260925 16:37:12.335076 21631 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4432fa4c6a9e41d4a2332ad8abb8f61f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4432fa4c6a9e41d4a2332ad8abb8f61f. Latest consensus state: current_term: 1 leader_uuid: "4432fa4c6a9e41d4a2332ad8abb8f61f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4432fa4c6a9e41d4a2332ad8abb8f61f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45217 } } }
I20260925 16:37:12.335166 21631 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4432fa4c6a9e41d4a2332ad8abb8f61f [sys.catalog]: This master's current role is: LEADER
I20260925 16:37:12.335258 21638 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:37:12.338515 21638 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:37:12.343901 21638 catalog_manager.cc:1347] Generated new cluster ID: 0aa802ec4f144f0cb2f16d11ebb094aa
I20260925 16:37:12.343998 21638 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:37:12.363497 21638 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:37:12.363816 21638 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:37:12.375761 21638 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4432fa4c6a9e41d4a2332ad8abb8f61f: Generated new TSK 0
I20260925 16:37:12.376500 21638 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:37:12.379276 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 21551
2026-09-25T16:37:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1387 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790353966814071-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:37:12 dist-test-slave-65pf krb5kdc[21650](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:37:12 dist-test-slave-65pf krb5kdc[21650](info): set up 2 sockets
Sep 25 16:37:12 dist-test-slave-65pf krb5kdc[21650](info): commencing operation
krb5kdc: starting...
W20260925 16:37:13.395104 392 mini_kdc.cc:121] Time spent starting KDC: real 0.996s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790353966814071-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790353966814071-392-0/krb5kdc/test-user.keytab.
Sep 25 16:37:13 dist-test-slave-65pf krb5kdc[21650](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:37:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:37:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790353966814071-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790353966814071-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 16:37:13.517630 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790353966814071-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790353966814071-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790353966814071-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46561
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790353966814071-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46561
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:37:13.624917 21663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:37:13.625233 21663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:37:13.625324 21663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:37:13.629376 21663 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:37:13.629475 21663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:37:13.629513 21663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:37:13.629550 21663 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:37:13.629582 21663 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:37:13.634639 21663 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790353966814071-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790353966814071-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46561
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790353966814071-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21663
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790353966814071-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:37:13.635667 21663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:37:13.636698 21663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:37:13.644889 21669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:37:13.644929 21671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:37:13.644946 21663 server_base.cc:1034] running on GCE node
W20260925 16:37:13.644934 21668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:37:13.645704 21663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:37:13.646804 21663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:37:13.647989 21663 hybrid_clock.cc:648] HybridClock initialized: now 1790354233647958 us; error 41 us; skew 500 ppm
Sep 25 16:37:13 dist-test-slave-65pf krb5kdc[21650](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354233, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:37:13.650758 21663 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 16:37:13.651652 21663 webserver.cc:460] Webserver started at http://127.0.0.1:46831/ using document root <none> and password file <none>
I20260925 16:37:13.652215 21663 fs_manager.cc:359] Metadata directory not provided
I20260925 16:37:13.652307 21663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:37:13.652526 21663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:37:13.654482 21663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790353966814071-392-0/minicluster-data/master-0/data/instance:
uuid: "c73a5b391e564518b02f2ff5d9721644"
format_stamp: "Formatted at 2026-09-25 16:37:13 on dist-test-slave-65pf"
server_key: "f914d3486de1a614c28badf50ae71f0d"
server_key_iv: "00543652b8f058b0bca239c54ecd2645"
server_key_version: "encryptionkey@0"
I20260925 16:37:13.655088 21663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790353966814071-392-0/minicluster-data/master-0/wal/instance:
uuid: "c73a5b391e564518b02f2ff5d9721644"
format_stamp: "Formatted at 2026-09-25 16:37:13 on dist-test-slave-65pf"
server_key: "f914d3486de1a614c28badf50ae71f0d"
server_key_iv: "00543652b8f058b0bca239c54ecd2645"
server_key_version: "encryptionkey@0"
I20260925 16:37:13.658741 21663 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 16:37:13.661204 21678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:37:13.662413 21663 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:37:13.662573 21663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790353966814071-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790353966814071-392-0/minicluster-data/master-0/wal
uuid: "c73a5b391e564518b02f2ff5d9721644"
format_stamp: "Formatted at 2026-09-25 16:37:13 on dist-test-slave-65pf"
server_key: "f914d3486de1a614c28badf50ae71f0d"
server_key_iv: "00543652b8f058b0bca239c54ecd2645"
server_key_version: "encryptionkey@0"
I20260925 16:37:13.662770 21663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790353966814071-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790353966814071-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790353966814071-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:37:13.680335 21663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:37:13.683759 21663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:37:13.683969 21663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:37:13.690886 21663 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46561
I20260925 16:37:13.690910 21730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46561 every 8 connection(s)
I20260925 16:37:13.691810 21663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:37:13.693212 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 21663
I20260925 16:37:13.693372 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790353966814071-392-0/minicluster-data/master-0/wal/instance
I20260925 16:37:13.693665 392 external_mini_cluster.cc:1428] Setting key d33ef96247cb8c3ee8a187df20cd3527
I20260925 16:37:13.694780 21731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:37:13.701099 21731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c73a5b391e564518b02f2ff5d9721644: Bootstrap starting.
Sep 25 16:37:13 dist-test-slave-65pf krb5kdc[21650](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 16:37:13.703805 21731 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c73a5b391e564518b02f2ff5d9721644: Neither blocks nor log segments found. Creating new log.
I20260925 16:37:13.704727 21731 log.cc:824] T 00000000000000000000000000000000 P c73a5b391e564518b02f2ff5d9721644: Log is configured to *not* fsync() on all Append() calls
I20260925 16:37:13.706856 21731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c73a5b391e564518b02f2ff5d9721644: No bootstrap required, opened a new log
I20260925 16:37:13.708765 21734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:13.695115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.106:35467 (local address 127.0.0.1:46561)
0925 16:37:13.695513 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:13.695531 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:13.695577 (+ 46us) server_negotiation.cc:407] Connection header received
0925 16:37:13.696231 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:13.696255 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:13.696613 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:13.696889 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:13.697860 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:13.698968 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:13.700162 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:13.700614 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:13.700736 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:13.703603 (+ 2867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:37:13.703609 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:37:13.703625 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:37:13.703656 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:37:13.705497 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:13.705995 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:13.706001 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:13.706006 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:13.706132 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:13.706426 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:13.706429 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:13.706431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:13.706949 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:37:13.707144 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:13.707423 (+ 279us) server_negotiation.cc:299] Negotiation successful
0925 16:37:13.707630 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":139,"threads_started":1}
I20260925 16:37:13.710573 21731 raft_consensus.cc:348] T 00000000000000000000000000000000 P c73a5b391e564518b02f2ff5d9721644 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c73a5b391e564518b02f2ff5d9721644" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46561 } }
I20260925 16:37:13.710826 21731 raft_consensus.cc:374] T 00000000000000000000000000000000 P c73a5b391e564518b02f2ff5d9721644 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:37:13.710913 21731 raft_consensus.cc:729] T 00000000000000000000000000000000 P c73a5b391e564518b02f2ff5d9721644 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c73a5b391e564518b02f2ff5d9721644, State: Initialized, Role: FOLLOWER
I20260925 16:37:13.711355 21731 consensus_queue.cc:260] T 00000000000000000000000000000000 P c73a5b391e564518b02f2ff5d9721644 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c73a5b391e564518b02f2ff5d9721644" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46561 } }
I20260925 16:37:13.711508 21731 raft_consensus.cc:388] T 00000000000000000000000000000000 P c73a5b391e564518b02f2ff5d9721644 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:37:13.711563 21731 raft_consensus.cc:482] T 00000000000000000000000000000000 P c73a5b391e564518b02f2ff5d9721644 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:37:13.711686 21731 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c73a5b391e564518b02f2ff5d9721644 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:37:13.711721 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:37:13.712478 21731 raft_consensus.cc:504] T 00000000000000000000000000000000 P c73a5b391e564518b02f2ff5d9721644 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c73a5b391e564518b02f2ff5d9721644" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46561 } }
I20260925 16:37:13.712809 21731 leader_election.cc:302] T 00000000000000000000000000000000 P c73a5b391e564518b02f2ff5d9721644 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c73a5b391e564518b02f2ff5d9721644; no voters:
I20260925 16:37:13.713104 21731 leader_election.cc:288] T 00000000000000000000000000000000 P c73a5b391e564518b02f2ff5d9721644 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:37:13.713213 21740 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c73a5b391e564518b02f2ff5d9721644 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:37:13.713443 21740 raft_consensus.cc:686] T 00000000000000000000000000000000 P c73a5b391e564518b02f2ff5d9721644 [term 1 LEADER]: Becoming Leader. State: Replica: c73a5b391e564518b02f2ff5d9721644, State: Running, Role: LEADER
I20260925 16:37:13.713721 21740 consensus_queue.cc:237] T 00000000000000000000000000000000 P c73a5b391e564518b02f2ff5d9721644 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c73a5b391e564518b02f2ff5d9721644" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46561 } }
I20260925 16:37:13.714313 21731 sys_catalog.cc:564] T 00000000000000000000000000000000 P c73a5b391e564518b02f2ff5d9721644 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:37:13.715509 21741 sys_catalog.cc:455] T 00000000000000000000000000000000 P c73a5b391e564518b02f2ff5d9721644 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c73a5b391e564518b02f2ff5d9721644" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c73a5b391e564518b02f2ff5d9721644" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46561 } } }
I20260925 16:37:13.715509 21742 sys_catalog.cc:455] T 00000000000000000000000000000000 P c73a5b391e564518b02f2ff5d9721644 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c73a5b391e564518b02f2ff5d9721644. Latest consensus state: current_term: 1 leader_uuid: "c73a5b391e564518b02f2ff5d9721644" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c73a5b391e564518b02f2ff5d9721644" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46561 } } }
I20260925 16:37:13.715648 21741 sys_catalog.cc:458] T 00000000000000000000000000000000 P c73a5b391e564518b02f2ff5d9721644 [sys.catalog]: This master's current role is: LEADER
I20260925 16:37:13.715734 21742 sys_catalog.cc:458] T 00000000000000000000000000000000 P c73a5b391e564518b02f2ff5d9721644 [sys.catalog]: This master's current role is: LEADER
I20260925 16:37:13.716601 21751 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:37:13.719362 21751 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:37:13.723186 21734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:13.713852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.106:39595 (local address 127.0.0.1:46561)
0925 16:37:13.713955 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:13.713959 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:13.714123 (+ 164us) server_negotiation.cc:407] Connection header received
0925 16:37:13.714240 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:13.714242 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:13.714285 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:13.714366 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:13.716282 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:13.717042 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:13.717859 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:13.718169 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:13.718271 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:13.719857 (+ 1586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:37:13.719860 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:37:13.719863 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:37:13.719886 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:37:13.721514 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:13.721965 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:13.721969 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:13.721970 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:13.722045 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:13.722364 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:13.722367 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:13.722369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:13.722523 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:37:13.722716 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:13.722955 (+ 239us) server_negotiation.cc:299] Negotiation successful
0925 16:37:13.723038 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 16:37:13.725123 21751 catalog_manager.cc:1347] Generated new cluster ID: 513c380a4fff4a11ad98d46ca4511ef6
I20260925 16:37:13.725193 21751 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:37:13.740478 21751 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:37:13.741415 21751 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:37:13.745219 21751 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c73a5b391e564518b02f2ff5d9721644: Generated new TSK 0
I20260925 16:37:13.745910 21751 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:37:13.753100 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 21663
W20260925 16:37:13.758384 21648 connection.cc:535] client connection to 127.0.0.1:46561 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46561): Connection reset by peer (error 104)
2026-09-25T16:37:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1373 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790353966814071-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:37:13 dist-test-slave-65pf krb5kdc[21762](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:37:13 dist-test-slave-65pf krb5kdc[21762](info): set up 2 sockets
Sep 25 16:37:13 dist-test-slave-65pf krb5kdc[21762](info): commencing operation
krb5kdc: starting...
W20260925 16:37:14.794819 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790353966814071-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790353966814071-392-0/krb5kdc/test-user.keytab.
Sep 25 16:37:14 dist-test-slave-65pf krb5kdc[21762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:37:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:37:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790353966814071-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790353966814071-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 16:37:14.914507 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790353966814071-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790353966814071-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790353966814071-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36863
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790353966814071-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36863
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:37:15.022790 21775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:37:15.023059 21775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:37:15.023102 21775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:37:15.027051 21775 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:37:15.027100 21775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:37:15.027119 21775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:37:15.027134 21775 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:37:15.027149 21775 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:37:15.032259 21775 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790353966814071-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790353966814071-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36863
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790353966814071-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21775
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790353966814071-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:37:15.033172 21775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:37:15.034083 21775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:37:15.042083 21781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:37:15.042099 21783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:37:15.042099 21780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:37:15.042559 21775 server_base.cc:1034] running on GCE node
I20260925 16:37:15.043063 21775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:37:15.044013 21775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:37:15.045181 21775 hybrid_clock.cc:648] HybridClock initialized: now 1790354235045173 us; error 28 us; skew 500 ppm
Sep 25 16:37:15 dist-test-slave-65pf krb5kdc[21762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354235, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:37:15.047876 21775 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 16:37:15.048745 21775 webserver.cc:460] Webserver started at http://127.0.0.1:41093/ using document root <none> and password file <none>
I20260925 16:37:15.049278 21775 fs_manager.cc:359] Metadata directory not provided
I20260925 16:37:15.049384 21775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:37:15.049605 21775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:37:15.051651 21775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790353966814071-392-0/minicluster-data/master-0/data/instance:
uuid: "92705b8b0a314a36bdbb99adc89c7c73"
format_stamp: "Formatted at 2026-09-25 16:37:15 on dist-test-slave-65pf"
server_key: "b298d824c5a0eee82ee163f8308fbca8"
server_key_iv: "da244ddf968a60300e22412c61e82576"
server_key_version: "encryptionkey@0"
I20260925 16:37:15.052206 21775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790353966814071-392-0/minicluster-data/master-0/wal/instance:
uuid: "92705b8b0a314a36bdbb99adc89c7c73"
format_stamp: "Formatted at 2026-09-25 16:37:15 on dist-test-slave-65pf"
server_key: "b298d824c5a0eee82ee163f8308fbca8"
server_key_iv: "da244ddf968a60300e22412c61e82576"
server_key_version: "encryptionkey@0"
I20260925 16:37:15.055980 21775 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 16:37:15.058372 21790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:37:15.059628 21775 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:37:15.059772 21775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790353966814071-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790353966814071-392-0/minicluster-data/master-0/wal
uuid: "92705b8b0a314a36bdbb99adc89c7c73"
format_stamp: "Formatted at 2026-09-25 16:37:15 on dist-test-slave-65pf"
server_key: "b298d824c5a0eee82ee163f8308fbca8"
server_key_iv: "da244ddf968a60300e22412c61e82576"
server_key_version: "encryptionkey@0"
I20260925 16:37:15.059890 21775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790353966814071-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790353966814071-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790353966814071-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:37:15.071171 21775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:37:15.074721 21775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:37:15.074927 21775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:37:15.082027 21775 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36863
I20260925 16:37:15.082052 21842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36863 every 8 connection(s)
I20260925 16:37:15.082965 21775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:37:15.085778 21843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:37:15.089995 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 21775
I20260925 16:37:15.090174 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790353966814071-392-0/minicluster-data/master-0/wal/instance
I20260925 16:37:15.090445 392 external_mini_cluster.cc:1428] Setting key 98b2f20eef8ac4c204cb49d21aa59682
I20260925 16:37:15.091812 21843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92705b8b0a314a36bdbb99adc89c7c73: Bootstrap starting.
I20260925 16:37:15.094250 21843 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92705b8b0a314a36bdbb99adc89c7c73: Neither blocks nor log segments found. Creating new log.
I20260925 16:37:15.095065 21843 log.cc:824] T 00000000000000000000000000000000 P 92705b8b0a314a36bdbb99adc89c7c73: Log is configured to *not* fsync() on all Append() calls
I20260925 16:37:15.097265 21843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92705b8b0a314a36bdbb99adc89c7c73: No bootstrap required, opened a new log
I20260925 16:37:15.099764 21846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:15.091949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.106:34519 (local address 127.0.0.1:36863)
0925 16:37:15.092355 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:15.092378 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:15.092413 (+ 35us) server_negotiation.cc:407] Connection header received
0925 16:37:15.093023 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:15.093040 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:15.093389 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:15.093663 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:15.094068 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:15.095137 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:15.096415 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:15.096874 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:15.096993 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:15.097928 (+ 935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:37:15.097932 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:37:15.097945 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:37:15.098306 (+ 361us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:37:15.098354 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:37:15.098385 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:37:15.098510 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:15.098771 (+ 261us) server_negotiation.cc:299] Negotiation successful
0925 16:37:15.098911 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":126,"threads_started":1}
I20260925 16:37:15.101368 21843 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92705b8b0a314a36bdbb99adc89c7c73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92705b8b0a314a36bdbb99adc89c7c73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36863 } }
I20260925 16:37:15.101655 21843 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92705b8b0a314a36bdbb99adc89c7c73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:37:15.101732 21843 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92705b8b0a314a36bdbb99adc89c7c73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92705b8b0a314a36bdbb99adc89c7c73, State: Initialized, Role: FOLLOWER
I20260925 16:37:15.102280 21843 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92705b8b0a314a36bdbb99adc89c7c73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92705b8b0a314a36bdbb99adc89c7c73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36863 } }
I20260925 16:37:15.102448 21843 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92705b8b0a314a36bdbb99adc89c7c73 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:37:15.102522 21843 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92705b8b0a314a36bdbb99adc89c7c73 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:37:15.102648 21843 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92705b8b0a314a36bdbb99adc89c7c73 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:37:15.102802 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:37:15.103837 21843 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92705b8b0a314a36bdbb99adc89c7c73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92705b8b0a314a36bdbb99adc89c7c73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36863 } }
I20260925 16:37:15.104226 21843 leader_election.cc:302] T 00000000000000000000000000000000 P 92705b8b0a314a36bdbb99adc89c7c73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92705b8b0a314a36bdbb99adc89c7c73; no voters:
I20260925 16:37:15.104574 21843 leader_election.cc:288] T 00000000000000000000000000000000 P 92705b8b0a314a36bdbb99adc89c7c73 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:37:15.104671 21852 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92705b8b0a314a36bdbb99adc89c7c73 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:37:15.105419 21852 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92705b8b0a314a36bdbb99adc89c7c73 [term 1 LEADER]: Becoming Leader. State: Replica: 92705b8b0a314a36bdbb99adc89c7c73, State: Running, Role: LEADER
I20260925 16:37:15.105804 21852 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92705b8b0a314a36bdbb99adc89c7c73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92705b8b0a314a36bdbb99adc89c7c73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36863 } }
I20260925 16:37:15.105970 21843 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92705b8b0a314a36bdbb99adc89c7c73 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:37:15.109036 21853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92705b8b0a314a36bdbb99adc89c7c73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "92705b8b0a314a36bdbb99adc89c7c73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92705b8b0a314a36bdbb99adc89c7c73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36863 } } }
I20260925 16:37:15.109170 21853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92705b8b0a314a36bdbb99adc89c7c73 [sys.catalog]: This master's current role is: LEADER
I20260925 16:37:15.109440 21855 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92705b8b0a314a36bdbb99adc89c7c73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92705b8b0a314a36bdbb99adc89c7c73. Latest consensus state: current_term: 1 leader_uuid: "92705b8b0a314a36bdbb99adc89c7c73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92705b8b0a314a36bdbb99adc89c7c73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36863 } } }
I20260925 16:37:15.109542 21855 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92705b8b0a314a36bdbb99adc89c7c73 [sys.catalog]: This master's current role is: LEADER
I20260925 16:37:15.110010 21846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:15.105215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.106:54515 (local address 127.0.0.1:36863)
0925 16:37:15.105338 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:15.105343 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:15.105609 (+ 266us) server_negotiation.cc:407] Connection header received
0925 16:37:15.105774 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:15.105781 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:15.105849 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:15.105941 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:15.106246 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:15.107020 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:15.107964 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:15.108272 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:15.108364 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:15.109410 (+ 1046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:37:15.109414 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:37:15.109416 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:37:15.109428 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:37:15.109451 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:37:15.109463 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:37:15.109538 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:15.109825 (+ 287us) server_negotiation.cc:299] Negotiation successful
0925 16:37:15.109856 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260925 16:37:15.111495 21867 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 92705b8b0a314a36bdbb99adc89c7c73: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 16:37:15.111595 21867 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 16:37:15.111681 21869 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:37:15.112356 21869 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:37:15.117157 21869 catalog_manager.cc:1347] Generated new cluster ID: e6fca6f027fa45baa8f3888d8bbb2049
I20260925 16:37:15.117219 21869 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:37:15.125980 21869 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:37:15.127241 21869 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:37:15.144513 21869 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 92705b8b0a314a36bdbb99adc89c7c73: Generated new TSK 0
I20260925 16:37:15.145287 21869 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:37:15.157986 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 21775
W20260925 16:37:15.163456 21760 connection.cc:535] client connection to 127.0.0.1:36863 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36863): Connection reset by peer (error 104)
2026-09-25T16:37:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1405 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790353966814071-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:37:15 dist-test-slave-65pf krb5kdc[21874](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:37:15 dist-test-slave-65pf krb5kdc[21874](info): set up 2 sockets
Sep 25 16:37:15 dist-test-slave-65pf krb5kdc[21874](info): commencing operation
krb5kdc: starting...
W20260925 16:37:16.178906 392 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790353966814071-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790353966814071-392-0/krb5kdc/test-user.keytab.
Sep 25 16:37:16 dist-test-slave-65pf krb5kdc[21874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354236, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:37:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:37:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790353966814071-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790353966814071-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 16:37:16.295799 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790353966814071-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790353966814071-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790353966814071-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33975
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790353966814071-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33975
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:37:16.404938 21887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:37:16.405182 21887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:37:16.405222 21887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:37:16.409068 21887 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:37:16.409119 21887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:37:16.409137 21887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:37:16.409152 21887 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:37:16.409168 21887 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:37:16.414177 21887 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790353966814071-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790353966814071-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33975
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790353966814071-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790353966814071-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:37:16.415109 21887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:37:16.416031 21887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:37:16.424201 21893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:37:16.424202 21892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:37:16.424260 21887 server_base.cc:1034] running on GCE node
W20260925 16:37:16.424202 21895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:37:16.424953 21887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:37:16.426015 21887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:37:16.427196 21887 hybrid_clock.cc:648] HybridClock initialized: now 1790354236427170 us; error 45 us; skew 500 ppm
Sep 25 16:37:16 dist-test-slave-65pf krb5kdc[21874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354236, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:37:16.439245 21887 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 16:37:16.440157 21887 webserver.cc:460] Webserver started at http://127.0.0.1:45663/ using document root <none> and password file <none>
I20260925 16:37:16.440732 21887 fs_manager.cc:359] Metadata directory not provided
I20260925 16:37:16.440819 21887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:37:16.441048 21887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:37:16.443130 21887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790353966814071-392-0/minicluster-data/master-0/data/instance:
uuid: "325e3a6c4c2f4122b04a37308e3a6dc0"
format_stamp: "Formatted at 2026-09-25 16:37:16 on dist-test-slave-65pf"
server_key: "05716d6f7090b7dee60b7d32048593af"
server_key_iv: "a881869b510833decb6c01508d1b70c7"
server_key_version: "encryptionkey@0"
I20260925 16:37:16.443670 21887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790353966814071-392-0/minicluster-data/master-0/wal/instance:
uuid: "325e3a6c4c2f4122b04a37308e3a6dc0"
format_stamp: "Formatted at 2026-09-25 16:37:16 on dist-test-slave-65pf"
server_key: "05716d6f7090b7dee60b7d32048593af"
server_key_iv: "a881869b510833decb6c01508d1b70c7"
server_key_version: "encryptionkey@0"
I20260925 16:37:16.447221 21887 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 16:37:16.449548 21902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:37:16.450831 21887 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:37:16.450961 21887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790353966814071-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790353966814071-392-0/minicluster-data/master-0/wal
uuid: "325e3a6c4c2f4122b04a37308e3a6dc0"
format_stamp: "Formatted at 2026-09-25 16:37:16 on dist-test-slave-65pf"
server_key: "05716d6f7090b7dee60b7d32048593af"
server_key_iv: "a881869b510833decb6c01508d1b70c7"
server_key_version: "encryptionkey@0"
I20260925 16:37:16.451072 21887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790353966814071-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790353966814071-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790353966814071-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:37:16.455550 21887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:37:16.458701 21887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:37:16.458885 21887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:37:16.465884 21887 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33975
I20260925 16:37:16.466032 21954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33975 every 8 connection(s)
I20260925 16:37:16.466835 21887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:37:16.469461 21955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:37:16.471252 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 21887
I20260925 16:37:16.471386 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790353966814071-392-0/minicluster-data/master-0/wal/instance
I20260925 16:37:16.471653 392 external_mini_cluster.cc:1428] Setting key 2f5b47455aba9df4cc2157182eafb985
I20260925 16:37:16.475834 21955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 325e3a6c4c2f4122b04a37308e3a6dc0: Bootstrap starting.
I20260925 16:37:16.476573 21958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:16.472948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.106:44263 (local address 127.0.0.1:33975)
0925 16:37:16.473352 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:16.473379 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:16.473423 (+ 44us) server_negotiation.cc:407] Connection header received
0925 16:37:16.474008 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:16.474023 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:16.474301 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:16.474566 (+ 265us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:16.475405 (+ 839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:37:16.475409 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:37:16.475421 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:37:16.475431 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:37:16.475478 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:37:16.475496 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:37:16.475600 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:16.475702 (+ 102us) server_negotiation.cc:299] Negotiation successful
0925 16:37:16.475867 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":121,"threads_started":1}
I20260925 16:37:16.478191 21955 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 325e3a6c4c2f4122b04a37308e3a6dc0: Neither blocks nor log segments found. Creating new log.
I20260925 16:37:16.479075 21955 log.cc:824] T 00000000000000000000000000000000 P 325e3a6c4c2f4122b04a37308e3a6dc0: Log is configured to *not* fsync() on all Append() calls
I20260925 16:37:16.479509 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:37:16.480909 21955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 325e3a6c4c2f4122b04a37308e3a6dc0: No bootstrap required, opened a new log
I20260925 16:37:16.482929 21958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:16.481469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.106:57475 (local address 127.0.0.1:33975)
0925 16:37:16.481584 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:16.481588 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:16.481759 (+ 171us) server_negotiation.cc:407] Connection header received
0925 16:37:16.481829 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:16.481832 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:16.481868 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:16.481940 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:16.482573 (+ 633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:37:16.482576 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:37:16.482593 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:37:16.482605 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:37:16.482630 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:37:16.482644 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:37:16.482734 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:16.482782 (+ 48us) server_negotiation.cc:299] Negotiation successful
0925 16:37:16.482816 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 16:37:16.484766 21955 raft_consensus.cc:348] T 00000000000000000000000000000000 P 325e3a6c4c2f4122b04a37308e3a6dc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "325e3a6c4c2f4122b04a37308e3a6dc0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33975 } }
I20260925 16:37:16.484969 21955 raft_consensus.cc:374] T 00000000000000000000000000000000 P 325e3a6c4c2f4122b04a37308e3a6dc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:37:16.485002 21955 raft_consensus.cc:729] T 00000000000000000000000000000000 P 325e3a6c4c2f4122b04a37308e3a6dc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 325e3a6c4c2f4122b04a37308e3a6dc0, State: Initialized, Role: FOLLOWER
I20260925 16:37:16.485332 21955 consensus_queue.cc:260] T 00000000000000000000000000000000 P 325e3a6c4c2f4122b04a37308e3a6dc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "325e3a6c4c2f4122b04a37308e3a6dc0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33975 } }
I20260925 16:37:16.485460 21955 raft_consensus.cc:388] T 00000000000000000000000000000000 P 325e3a6c4c2f4122b04a37308e3a6dc0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:37:16.485502 21955 raft_consensus.cc:482] T 00000000000000000000000000000000 P 325e3a6c4c2f4122b04a37308e3a6dc0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:37:16.485570 21955 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 325e3a6c4c2f4122b04a37308e3a6dc0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:37:16.486336 21955 raft_consensus.cc:504] T 00000000000000000000000000000000 P 325e3a6c4c2f4122b04a37308e3a6dc0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "325e3a6c4c2f4122b04a37308e3a6dc0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33975 } }
I20260925 16:37:16.486686 21955 leader_election.cc:302] T 00000000000000000000000000000000 P 325e3a6c4c2f4122b04a37308e3a6dc0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 325e3a6c4c2f4122b04a37308e3a6dc0; no voters:
I20260925 16:37:16.486927 21955 leader_election.cc:288] T 00000000000000000000000000000000 P 325e3a6c4c2f4122b04a37308e3a6dc0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:37:16.487030 21965 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 325e3a6c4c2f4122b04a37308e3a6dc0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:37:16.487277 21965 raft_consensus.cc:686] T 00000000000000000000000000000000 P 325e3a6c4c2f4122b04a37308e3a6dc0 [term 1 LEADER]: Becoming Leader. State: Replica: 325e3a6c4c2f4122b04a37308e3a6dc0, State: Running, Role: LEADER
I20260925 16:37:16.487609 21965 consensus_queue.cc:237] T 00000000000000000000000000000000 P 325e3a6c4c2f4122b04a37308e3a6dc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "325e3a6c4c2f4122b04a37308e3a6dc0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33975 } }
I20260925 16:37:16.488114 21955 sys_catalog.cc:564] T 00000000000000000000000000000000 P 325e3a6c4c2f4122b04a37308e3a6dc0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:37:16.489372 21967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 325e3a6c4c2f4122b04a37308e3a6dc0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 325e3a6c4c2f4122b04a37308e3a6dc0. Latest consensus state: current_term: 1 leader_uuid: "325e3a6c4c2f4122b04a37308e3a6dc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "325e3a6c4c2f4122b04a37308e3a6dc0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33975 } } }
I20260925 16:37:16.489398 21966 sys_catalog.cc:455] T 00000000000000000000000000000000 P 325e3a6c4c2f4122b04a37308e3a6dc0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "325e3a6c4c2f4122b04a37308e3a6dc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "325e3a6c4c2f4122b04a37308e3a6dc0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33975 } } }
I20260925 16:37:16.489530 21967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 325e3a6c4c2f4122b04a37308e3a6dc0 [sys.catalog]: This master's current role is: LEADER
I20260925 16:37:16.489544 21966 sys_catalog.cc:458] T 00000000000000000000000000000000 P 325e3a6c4c2f4122b04a37308e3a6dc0 [sys.catalog]: This master's current role is: LEADER
I20260925 16:37:16.490022 21976 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:37:16.492558 21976 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:37:16.497555 21976 catalog_manager.cc:1347] Generated new cluster ID: bdf506e4c8024eea8cf2f00d9d64d4bd
I20260925 16:37:16.497622 21976 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:37:16.517766 21976 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:37:16.518117 21976 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:37:16.526255 21976 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 325e3a6c4c2f4122b04a37308e3a6dc0: Generated new TSK 0
I20260925 16:37:16.526983 21976 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:37:16.539687 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 21887
2026-09-25T16:37:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1383 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790353966814071-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:37:16 dist-test-slave-65pf krb5kdc[21986](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:37:16 dist-test-slave-65pf krb5kdc[21986](info): set up 2 sockets
Sep 25 16:37:16 dist-test-slave-65pf krb5kdc[21986](info): commencing operation
krb5kdc: starting...
W20260925 16:37:17.574931 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790353966814071-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790353966814071-392-0/krb5kdc/test-user.keytab.
Sep 25 16:37:17 dist-test-slave-65pf krb5kdc[21986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:37:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:37:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 16:37:17.689955 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790353966814071-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790353966814071-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790353966814071-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37983
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37983
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:37:17.798022 21999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:37:17.798297 21999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:37:17.798369 21999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:37:17.802258 21999 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:37:17.802345 21999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:37:17.802376 21999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:37:17.802414 21999 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:37:17.802449 21999 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:37:17.807637 21999 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790353966814071-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790353966814071-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37983
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790353966814071-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:37:17.808704 21999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:37:17.809682 21999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:37:17.817826 22005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:37:17.817860 22007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:37:17.817831 22004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:37:17.817996 21999 server_base.cc:1034] running on GCE node
I20260925 16:37:17.818661 21999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:37:17.819715 21999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:37:17.820901 21999 hybrid_clock.cc:648] HybridClock initialized: now 1790354237820888 us; error 27 us; skew 500 ppm
Sep 25 16:37:17 dist-test-slave-65pf krb5kdc[21986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354237, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:37:17.823709 21999 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 16:37:17.824563 21999 webserver.cc:460] Webserver started at http://127.0.98.62:42919/ using document root <none> and password file <none>
I20260925 16:37:17.825139 21999 fs_manager.cc:359] Metadata directory not provided
I20260925 16:37:17.825229 21999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:37:17.825469 21999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:37:17.827492 21999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790353966814071-392-0/minicluster-data/master-0/data/instance:
uuid: "ff9bd855a81442cea365f019f27ef715"
format_stamp: "Formatted at 2026-09-25 16:37:17 on dist-test-slave-65pf"
server_key: "ca4d4b8ba0cc1608b2d58b0cb0c817d0"
server_key_iv: "df0268b726de7afc2f324603eb13d912"
server_key_version: "encryptionkey@0"
I20260925 16:37:17.828013 21999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790353966814071-392-0/minicluster-data/master-0/wal/instance:
uuid: "ff9bd855a81442cea365f019f27ef715"
format_stamp: "Formatted at 2026-09-25 16:37:17 on dist-test-slave-65pf"
server_key: "ca4d4b8ba0cc1608b2d58b0cb0c817d0"
server_key_iv: "df0268b726de7afc2f324603eb13d912"
server_key_version: "encryptionkey@0"
I20260925 16:37:17.831465 21999 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 16:37:17.833674 22014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:37:17.834939 21999 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:37:17.835076 21999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790353966814071-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790353966814071-392-0/minicluster-data/master-0/wal
uuid: "ff9bd855a81442cea365f019f27ef715"
format_stamp: "Formatted at 2026-09-25 16:37:17 on dist-test-slave-65pf"
server_key: "ca4d4b8ba0cc1608b2d58b0cb0c817d0"
server_key_iv: "df0268b726de7afc2f324603eb13d912"
server_key_version: "encryptionkey@0"
I20260925 16:37:17.835179 21999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790353966814071-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790353966814071-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790353966814071-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:37:17.850411 21999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:37:17.853652 21999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:37:17.853878 21999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:37:17.860918 21999 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37983
I20260925 16:37:17.860934 22066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37983 every 8 connection(s)
I20260925 16:37:17.861919 21999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:37:17.864827 22067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:37:17.865500 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 21999
I20260925 16:37:17.865614 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790353966814071-392-0/minicluster-data/master-0/wal/instance
I20260925 16:37:17.865926 392 external_mini_cluster.cc:1428] Setting key e06761a18ae63c2298ffa1269ae23dfa
I20260925 16:37:17.872040 22067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff9bd855a81442cea365f019f27ef715: Bootstrap starting.
I20260925 16:37:17.874852 22067 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff9bd855a81442cea365f019f27ef715: Neither blocks nor log segments found. Creating new log.
I20260925 16:37:17.875806 22067 log.cc:824] T 00000000000000000000000000000000 P ff9bd855a81442cea365f019f27ef715: Log is configured to *not* fsync() on all Append() calls
Sep 25 16:37:17 dist-test-slave-65pf krb5kdc[21986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:37:17.877983 22067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff9bd855a81442cea365f019f27ef715: No bootstrap required, opened a new log
I20260925 16:37:17.881610 22070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:17.867414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42927 (local address 127.0.98.62:37983)
0925 16:37:17.867854 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:17.867878 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:17.867926 (+ 48us) server_negotiation.cc:407] Connection header received
0925 16:37:17.868829 (+ 903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:17.868853 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:17.869290 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:17.869683 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:17.870338 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:17.871874 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:17.872816 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:17.873523 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:17.873713 (+ 190us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:17.876620 (+ 2907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:37:17.876629 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:37:17.876649 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:37:17.876681 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:37:17.878504 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:17.879183 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:17.879188 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:17.879193 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:17.879297 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:17.879658 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:17.879660 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:17.879661 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:17.880043 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:37:17.880197 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:17.880620 (+ 423us) server_negotiation.cc:299] Negotiation successful
0925 16:37:17.880841 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":96,"threads_started":1}
I20260925 16:37:17.882318 22067 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff9bd855a81442cea365f019f27ef715 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff9bd855a81442cea365f019f27ef715" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37983 } }
I20260925 16:37:17.882582 22067 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff9bd855a81442cea365f019f27ef715 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:37:17.882690 22067 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff9bd855a81442cea365f019f27ef715 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff9bd855a81442cea365f019f27ef715, State: Initialized, Role: FOLLOWER
I20260925 16:37:17.883160 22067 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff9bd855a81442cea365f019f27ef715 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff9bd855a81442cea365f019f27ef715" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37983 } }
I20260925 16:37:17.883337 22067 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff9bd855a81442cea365f019f27ef715 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:37:17.883427 22067 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff9bd855a81442cea365f019f27ef715 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:37:17.883545 22067 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff9bd855a81442cea365f019f27ef715 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:37:17.884781 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:37:17.884763 22067 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff9bd855a81442cea365f019f27ef715 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff9bd855a81442cea365f019f27ef715" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37983 } }
I20260925 16:37:17.885106 22067 leader_election.cc:302] T 00000000000000000000000000000000 P ff9bd855a81442cea365f019f27ef715 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff9bd855a81442cea365f019f27ef715; no voters:
I20260925 16:37:17.885497 22067 leader_election.cc:288] T 00000000000000000000000000000000 P ff9bd855a81442cea365f019f27ef715 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:37:17.885578 22072 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff9bd855a81442cea365f019f27ef715 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:37:17.885761 22072 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff9bd855a81442cea365f019f27ef715 [term 1 LEADER]: Becoming Leader. State: Replica: ff9bd855a81442cea365f019f27ef715, State: Running, Role: LEADER
I20260925 16:37:17.886125 22072 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff9bd855a81442cea365f019f27ef715 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff9bd855a81442cea365f019f27ef715" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37983 } }
I20260925 16:37:17.886917 22067 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff9bd855a81442cea365f019f27ef715 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:37:17.887902 22073 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff9bd855a81442cea365f019f27ef715 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff9bd855a81442cea365f019f27ef715" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff9bd855a81442cea365f019f27ef715" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37983 } } }
I20260925 16:37:17.888032 22073 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff9bd855a81442cea365f019f27ef715 [sys.catalog]: This master's current role is: LEADER
I20260925 16:37:17.888305 22074 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff9bd855a81442cea365f019f27ef715 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff9bd855a81442cea365f019f27ef715. Latest consensus state: current_term: 1 leader_uuid: "ff9bd855a81442cea365f019f27ef715" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff9bd855a81442cea365f019f27ef715" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37983 } } }
I20260925 16:37:17.888439 22074 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff9bd855a81442cea365f019f27ef715 [sys.catalog]: This master's current role is: LEADER
W20260925 16:37:17.891979 22093 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ff9bd855a81442cea365f019f27ef715: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 16:37:17.892063 22093 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 16:37:17.892149 22085 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:37:17.893146 22085 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:37:17.897641 22070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:17.887847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56391 (local address 127.0.98.62:37983)
0925 16:37:17.888606 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:17.888610 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:17.888631 (+ 21us) server_negotiation.cc:407] Connection header received
0925 16:37:17.888671 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:17.888674 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:17.888718 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:17.888798 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:17.889994 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:17.890781 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:17.891903 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:17.892265 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:17.892382 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:17.894125 (+ 1743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:37:17.894129 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:37:17.894132 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:37:17.894171 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:37:17.895841 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:17.896357 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:17.896361 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:17.896363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:17.896445 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:17.896781 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:17.896785 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:17.896786 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:17.896964 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:37:17.897118 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:17.897384 (+ 266us) server_negotiation.cc:299] Negotiation successful
0925 16:37:17.897478 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":692}
I20260925 16:37:17.902315 22085 catalog_manager.cc:1347] Generated new cluster ID: 4ab4f0156043484f8bbc6983272e3624
I20260925 16:37:17.902428 22085 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:37:17.928431 22085 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:37:17.929560 22085 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:37:17.935680 22085 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff9bd855a81442cea365f019f27ef715: Generated new TSK 0
I20260925 16:37:17.936389 22085 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:37:17.950747 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 21999
W20260925 16:37:17.956158 21984 connection.cc:535] client connection to 127.0.98.62:37983 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:37983): Connection reset by peer (error 104)
2026-09-25T16:37:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1409 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790353966814071-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:37:18 dist-test-slave-65pf krb5kdc[22098](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:37:18 dist-test-slave-65pf krb5kdc[22098](info): set up 2 sockets
Sep 25 16:37:18 dist-test-slave-65pf krb5kdc[22098](info): commencing operation
krb5kdc: starting...
W20260925 16:37:18.982937 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790353966814071-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790353966814071-392-0/krb5kdc/test-user.keytab.
Sep 25 16:37:19 dist-test-slave-65pf krb5kdc[22098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354239, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:37:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:37:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 16:37:19.101451 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790353966814071-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790353966814071-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790353966814071-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42925
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42925
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:37:19.208866 22111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:37:19.209231 22111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:37:19.209281 22111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:37:19.213231 22111 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:37:19.213281 22111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:37:19.213300 22111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:37:19.213315 22111 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:37:19.213330 22111 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:37:19.218416 22111 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790353966814071-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790353966814071-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42925
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790353966814071-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:37:19.219441 22111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:37:19.220397 22111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:37:19.228597 22119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:37:19.228751 22111 server_base.cc:1034] running on GCE node
W20260925 16:37:19.228667 22117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:37:19.228597 22116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:37:19.229460 22111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:37:19.230523 22111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:37:19.231674 22111 hybrid_clock.cc:648] HybridClock initialized: now 1790354239231652 us; error 52 us; skew 500 ppm
Sep 25 16:37:19 dist-test-slave-65pf krb5kdc[22098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354239, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:37:19.234416 22111 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 16:37:19.235347 22111 webserver.cc:460] Webserver started at http://127.0.98.62:44407/ using document root <none> and password file <none>
I20260925 16:37:19.235903 22111 fs_manager.cc:359] Metadata directory not provided
I20260925 16:37:19.235992 22111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:37:19.236228 22111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:37:19.238145 22111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790353966814071-392-0/minicluster-data/master-0/data/instance:
uuid: "4de2feea4ec64beb9f2468f7e214cc0d"
format_stamp: "Formatted at 2026-09-25 16:37:19 on dist-test-slave-65pf"
server_key: "d7c72733f68b5dffe313659f176168bd"
server_key_iv: "6c9f5b28bde8dacf7ed5153ab9a3315c"
server_key_version: "encryptionkey@0"
I20260925 16:37:19.238772 22111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790353966814071-392-0/minicluster-data/master-0/wal/instance:
uuid: "4de2feea4ec64beb9f2468f7e214cc0d"
format_stamp: "Formatted at 2026-09-25 16:37:19 on dist-test-slave-65pf"
server_key: "d7c72733f68b5dffe313659f176168bd"
server_key_iv: "6c9f5b28bde8dacf7ed5153ab9a3315c"
server_key_version: "encryptionkey@0"
I20260925 16:37:19.242405 22111 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 16:37:19.244876 22126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:37:19.246162 22111 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:37:19.246302 22111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790353966814071-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790353966814071-392-0/minicluster-data/master-0/wal
uuid: "4de2feea4ec64beb9f2468f7e214cc0d"
format_stamp: "Formatted at 2026-09-25 16:37:19 on dist-test-slave-65pf"
server_key: "d7c72733f68b5dffe313659f176168bd"
server_key_iv: "6c9f5b28bde8dacf7ed5153ab9a3315c"
server_key_version: "encryptionkey@0"
I20260925 16:37:19.246421 22111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790353966814071-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790353966814071-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790353966814071-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:37:19.288439 22111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:37:19.291993 22111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:37:19.292202 22111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:37:19.299311 22111 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42925
I20260925 16:37:19.299335 22178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42925 every 8 connection(s)
I20260925 16:37:19.300282 22111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:37:19.303004 22179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:37:19.307819 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 22111
I20260925 16:37:19.307986 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790353966814071-392-0/minicluster-data/master-0/wal/instance
I20260925 16:37:19.308297 392 external_mini_cluster.cc:1428] Setting key fded0d19dca177d5c9394fb53d4b4297
I20260925 16:37:19.308674 22179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4de2feea4ec64beb9f2468f7e214cc0d: Bootstrap starting.
I20260925 16:37:19.311299 22179 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4de2feea4ec64beb9f2468f7e214cc0d: Neither blocks nor log segments found. Creating new log.
I20260925 16:37:19.312019 22179 log.cc:824] T 00000000000000000000000000000000 P 4de2feea4ec64beb9f2468f7e214cc0d: Log is configured to *not* fsync() on all Append() calls
I20260925 16:37:19.313809 22179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4de2feea4ec64beb9f2468f7e214cc0d: No bootstrap required, opened a new log
I20260925 16:37:19.317299 22179 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4de2feea4ec64beb9f2468f7e214cc0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de2feea4ec64beb9f2468f7e214cc0d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42925 } }
I20260925 16:37:19.317538 22179 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4de2feea4ec64beb9f2468f7e214cc0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:37:19.317615 22179 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4de2feea4ec64beb9f2468f7e214cc0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4de2feea4ec64beb9f2468f7e214cc0d, State: Initialized, Role: FOLLOWER
I20260925 16:37:19.317682 22182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:19.309801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58005 (local address 127.0.98.62:42925)
0925 16:37:19.310236 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:19.310250 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:19.310284 (+ 34us) server_negotiation.cc:407] Connection header received
0925 16:37:19.311217 (+ 933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:19.311236 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:19.311633 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:19.312007 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:19.312405 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:19.313335 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:19.314269 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:19.314838 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:19.315046 (+ 208us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:19.315937 (+ 891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:37:19.315947 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:37:19.315959 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:37:19.316341 (+ 382us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:37:19.316392 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:37:19.316414 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:37:19.316562 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:19.316773 (+ 211us) server_negotiation.cc:299] Negotiation successful
0925 16:37:19.316912 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":121,"threads_started":1}
I20260925 16:37:19.318127 22179 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4de2feea4ec64beb9f2468f7e214cc0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de2feea4ec64beb9f2468f7e214cc0d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42925 } }
I20260925 16:37:19.318291 22179 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4de2feea4ec64beb9f2468f7e214cc0d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:37:19.318380 22179 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4de2feea4ec64beb9f2468f7e214cc0d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:37:19.318477 22179 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4de2feea4ec64beb9f2468f7e214cc0d [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:37:19.319535 22179 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4de2feea4ec64beb9f2468f7e214cc0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de2feea4ec64beb9f2468f7e214cc0d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42925 } }
I20260925 16:37:19.319928 22179 leader_election.cc:302] T 00000000000000000000000000000000 P 4de2feea4ec64beb9f2468f7e214cc0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4de2feea4ec64beb9f2468f7e214cc0d; no voters:
I20260925 16:37:19.320267 22179 leader_election.cc:288] T 00000000000000000000000000000000 P 4de2feea4ec64beb9f2468f7e214cc0d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:37:19.320340 22184 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4de2feea4ec64beb9f2468f7e214cc0d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:37:19.320556 22184 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4de2feea4ec64beb9f2468f7e214cc0d [term 1 LEADER]: Becoming Leader. State: Replica: 4de2feea4ec64beb9f2468f7e214cc0d, State: Running, Role: LEADER
I20260925 16:37:19.320708 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:37:19.320910 22184 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4de2feea4ec64beb9f2468f7e214cc0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de2feea4ec64beb9f2468f7e214cc0d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42925 } }
I20260925 16:37:19.321574 22179 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4de2feea4ec64beb9f2468f7e214cc0d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:37:19.322309 22185 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4de2feea4ec64beb9f2468f7e214cc0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4de2feea4ec64beb9f2468f7e214cc0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de2feea4ec64beb9f2468f7e214cc0d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42925 } } }
I20260925 16:37:19.322415 22185 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4de2feea4ec64beb9f2468f7e214cc0d [sys.catalog]: This master's current role is: LEADER
I20260925 16:37:19.322736 22186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4de2feea4ec64beb9f2468f7e214cc0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4de2feea4ec64beb9f2468f7e214cc0d. Latest consensus state: current_term: 1 leader_uuid: "4de2feea4ec64beb9f2468f7e214cc0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de2feea4ec64beb9f2468f7e214cc0d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42925 } } }
I20260925 16:37:19.322829 22186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4de2feea4ec64beb9f2468f7e214cc0d [sys.catalog]: This master's current role is: LEADER
I20260925 16:37:19.323060 22197 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:37:19.326308 22197 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:37:19.328858 22182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:19.324236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55469 (local address 127.0.98.62:42925)
0925 16:37:19.324362 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:19.324367 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:19.324592 (+ 225us) server_negotiation.cc:407] Connection header received
0925 16:37:19.324708 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:19.324710 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:19.324753 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:19.324838 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:19.325114 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:19.325884 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:19.327047 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:19.327376 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:19.327477 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:19.328381 (+ 904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:37:19.328384 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:37:19.328387 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:37:19.328401 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:37:19.328428 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:37:19.328440 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:37:19.328559 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:19.328686 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 16:37:19.328718 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 16:37:19.332412 22197 catalog_manager.cc:1347] Generated new cluster ID: 4cc0dadcc9c84d328851790587eff19b
I20260925 16:37:19.332489 22197 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:37:19.354326 22197 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:37:19.355355 22197 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:37:19.362319 22197 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4de2feea4ec64beb9f2468f7e214cc0d: Generated new TSK 0
I20260925 16:37:19.363061 22197 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:37:19.375216 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 22111
W20260925 16:37:19.380918 22096 connection.cc:535] client connection to 127.0.98.62:42925 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:42925): Connection reset by peer (error 104)
2026-09-25T16:37:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1424 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790353966814071-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:37:19 dist-test-slave-65pf krb5kdc[22210](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:37:19 dist-test-slave-65pf krb5kdc[22210](info): set up 2 sockets
Sep 25 16:37:19 dist-test-slave-65pf krb5kdc[22210](info): commencing operation
krb5kdc: starting...
W20260925 16:37:20.402966 392 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790353966814071-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790353966814071-392-0/krb5kdc/test-user.keytab.
Sep 25 16:37:20 dist-test-slave-65pf krb5kdc[22210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:37:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:37:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 16:37:20.521019 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790353966814071-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790353966814071-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790353966814071-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39265
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39265
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:37:20.628670 22223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:37:20.628947 22223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:37:20.628994 22223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:37:20.633077 22223 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:37:20.633136 22223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:37:20.633157 22223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:37:20.633178 22223 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:37:20.633193 22223 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:37:20.638937 22223 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790353966814071-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790353966814071-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39265
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790353966814071-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:37:20.639922 22223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:37:20.640862 22223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:37:20.649307 22229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:37:20.649310 22228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:37:20.649566 22223 server_base.cc:1034] running on GCE node
W20260925 16:37:20.649310 22231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:37:20.650244 22223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:37:20.651422 22223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:37:20.652580 22223 hybrid_clock.cc:648] HybridClock initialized: now 1790354240652548 us; error 44 us; skew 500 ppm
Sep 25 16:37:20 dist-test-slave-65pf krb5kdc[22210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354240, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:37:20.655256 22223 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 16:37:20.656172 22223 webserver.cc:460] Webserver started at http://127.0.98.62:38293/ using document root <none> and password file <none>
I20260925 16:37:20.656723 22223 fs_manager.cc:359] Metadata directory not provided
I20260925 16:37:20.656819 22223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:37:20.657035 22223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:37:20.659066 22223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790353966814071-392-0/minicluster-data/master-0/data/instance:
uuid: "7dbdb2520da34b36a3afd133ed615e17"
format_stamp: "Formatted at 2026-09-25 16:37:20 on dist-test-slave-65pf"
server_key: "ede82e47153243dfcb0b834f040e9fe6"
server_key_iv: "6934eb6a03561774a703a158e914160a"
server_key_version: "encryptionkey@0"
I20260925 16:37:20.659641 22223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790353966814071-392-0/minicluster-data/master-0/wal/instance:
uuid: "7dbdb2520da34b36a3afd133ed615e17"
format_stamp: "Formatted at 2026-09-25 16:37:20 on dist-test-slave-65pf"
server_key: "ede82e47153243dfcb0b834f040e9fe6"
server_key_iv: "6934eb6a03561774a703a158e914160a"
server_key_version: "encryptionkey@0"
I20260925 16:37:20.663306 22223 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 16:37:20.665647 22238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:37:20.666919 22223 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:37:20.667050 22223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790353966814071-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790353966814071-392-0/minicluster-data/master-0/wal
uuid: "7dbdb2520da34b36a3afd133ed615e17"
format_stamp: "Formatted at 2026-09-25 16:37:20 on dist-test-slave-65pf"
server_key: "ede82e47153243dfcb0b834f040e9fe6"
server_key_iv: "6934eb6a03561774a703a158e914160a"
server_key_version: "encryptionkey@0"
I20260925 16:37:20.667181 22223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790353966814071-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790353966814071-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790353966814071-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:37:20.671736 22223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:37:20.674419 22223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:37:20.674672 22223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:37:20.681898 22223 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39265
I20260925 16:37:20.681917 22290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39265 every 8 connection(s)
I20260925 16:37:20.682884 22223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:37:20.685492 22291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:37:20.687114 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 22223
I20260925 16:37:20.687271 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790353966814071-392-0/minicluster-data/master-0/wal/instance
I20260925 16:37:20.687544 392 external_mini_cluster.cc:1428] Setting key c7c2046d3f1869f5e121a9652e24b5cc
I20260925 16:37:20.691167 22291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dbdb2520da34b36a3afd133ed615e17: Bootstrap starting.
I20260925 16:37:20.692956 22294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:20.688845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51967 (local address 127.0.98.62:39265)
0925 16:37:20.689229 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:20.689249 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:20.689477 (+ 228us) server_negotiation.cc:407] Connection header received
0925 16:37:20.690343 (+ 866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:20.690361 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:20.690755 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:20.691082 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:20.691812 (+ 730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:37:20.691816 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:37:20.691840 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:37:20.691847 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:37:20.691888 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:37:20.691905 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:37:20.692007 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:20.692099 (+ 92us) server_negotiation.cc:299] Negotiation successful
0925 16:37:20.692261 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":139,"threads_started":1}
I20260925 16:37:20.693609 22291 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7dbdb2520da34b36a3afd133ed615e17: Neither blocks nor log segments found. Creating new log.
I20260925 16:37:20.694339 22291 log.cc:824] T 00000000000000000000000000000000 P 7dbdb2520da34b36a3afd133ed615e17: Log is configured to *not* fsync() on all Append() calls
I20260925 16:37:20.695930 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:37:20.696388 22291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dbdb2520da34b36a3afd133ed615e17: No bootstrap required, opened a new log
I20260925 16:37:20.699450 22294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:20.697887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33241 (local address 127.0.98.62:39265)
0925 16:37:20.697993 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:20.697998 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:20.698166 (+ 168us) server_negotiation.cc:407] Connection header received
0925 16:37:20.698307 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:20.698309 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:20.698333 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:20.698394 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:20.699081 (+ 687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:37:20.699083 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:37:20.699085 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:37:20.699094 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:37:20.699115 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:37:20.699129 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:37:20.699194 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:20.699300 (+ 106us) server_negotiation.cc:299] Negotiation successful
0925 16:37:20.699327 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 16:37:20.700525 22291 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7dbdb2520da34b36a3afd133ed615e17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dbdb2520da34b36a3afd133ed615e17" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39265 } }
I20260925 16:37:20.700775 22291 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7dbdb2520da34b36a3afd133ed615e17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:37:20.700841 22291 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7dbdb2520da34b36a3afd133ed615e17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7dbdb2520da34b36a3afd133ed615e17, State: Initialized, Role: FOLLOWER
I20260925 16:37:20.701220 22291 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7dbdb2520da34b36a3afd133ed615e17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dbdb2520da34b36a3afd133ed615e17" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39265 } }
I20260925 16:37:20.701375 22291 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7dbdb2520da34b36a3afd133ed615e17 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:37:20.701460 22291 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7dbdb2520da34b36a3afd133ed615e17 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:37:20.701555 22291 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7dbdb2520da34b36a3afd133ed615e17 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:37:20.702392 22291 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7dbdb2520da34b36a3afd133ed615e17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dbdb2520da34b36a3afd133ed615e17" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39265 } }
I20260925 16:37:20.702764 22291 leader_election.cc:302] T 00000000000000000000000000000000 P 7dbdb2520da34b36a3afd133ed615e17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7dbdb2520da34b36a3afd133ed615e17; no voters:
I20260925 16:37:20.703042 22291 leader_election.cc:288] T 00000000000000000000000000000000 P 7dbdb2520da34b36a3afd133ed615e17 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:37:20.703140 22301 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7dbdb2520da34b36a3afd133ed615e17 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:37:20.703408 22301 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7dbdb2520da34b36a3afd133ed615e17 [term 1 LEADER]: Becoming Leader. State: Replica: 7dbdb2520da34b36a3afd133ed615e17, State: Running, Role: LEADER
I20260925 16:37:20.703754 22301 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7dbdb2520da34b36a3afd133ed615e17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dbdb2520da34b36a3afd133ed615e17" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39265 } }
I20260925 16:37:20.704257 22291 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7dbdb2520da34b36a3afd133ed615e17 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:37:20.704828 22303 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dbdb2520da34b36a3afd133ed615e17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7dbdb2520da34b36a3afd133ed615e17. Latest consensus state: current_term: 1 leader_uuid: "7dbdb2520da34b36a3afd133ed615e17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dbdb2520da34b36a3afd133ed615e17" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39265 } } }
I20260925 16:37:20.704964 22303 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dbdb2520da34b36a3afd133ed615e17 [sys.catalog]: This master's current role is: LEADER
I20260925 16:37:20.705312 22302 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dbdb2520da34b36a3afd133ed615e17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7dbdb2520da34b36a3afd133ed615e17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dbdb2520da34b36a3afd133ed615e17" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39265 } } }
I20260925 16:37:20.705433 22302 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dbdb2520da34b36a3afd133ed615e17 [sys.catalog]: This master's current role is: LEADER
I20260925 16:37:20.705678 22307 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:37:20.708669 22307 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:37:20.714257 22307 catalog_manager.cc:1347] Generated new cluster ID: 4e283067cc9d467eb8045faae8dde5ff
I20260925 16:37:20.714316 22307 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:37:20.726476 22307 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:37:20.726815 22307 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:37:20.736243 22307 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7dbdb2520da34b36a3afd133ed615e17: Generated new TSK 0
I20260925 16:37:20.737002 22307 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:37:20.747586 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 22223
2026-09-25T16:37:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1371 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790353966814071-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:37:20 dist-test-slave-65pf krb5kdc[22322](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:37:20 dist-test-slave-65pf krb5kdc[22322](info): set up 2 sockets
Sep 25 16:37:20 dist-test-slave-65pf krb5kdc[22322](info): commencing operation
krb5kdc: starting...
W20260925 16:37:21.767180 392 mini_kdc.cc:121] Time spent starting KDC: real 1.000s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790353966814071-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790353966814071-392-0/krb5kdc/test-user.keytab.
Sep 25 16:37:21 dist-test-slave-65pf krb5kdc[22322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354241, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:37:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:37:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 16:37:21.879292 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790353966814071-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790353966814071-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790353966814071-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39379
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39379
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:37:21.983395 22335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:37:21.983676 22335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:37:21.983747 22335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:37:21.988039 22335 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:37:21.988132 22335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:37:21.988166 22335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:37:21.988209 22335 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:37:21.988240 22335 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:37:21.993418 22335 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790353966814071-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790353966814071-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39379
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790353966814071-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:37:21.994673 22335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:37:21.996223 22335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:37:22.004603 22341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:37:22.004603 22343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:37:22.004599 22340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:37:22.004997 22335 server_base.cc:1034] running on GCE node
I20260925 16:37:22.005509 22335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:37:22.006680 22335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:37:22.007886 22335 hybrid_clock.cc:648] HybridClock initialized: now 1790354242007873 us; error 27 us; skew 500 ppm
Sep 25 16:37:22 dist-test-slave-65pf krb5kdc[22322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354242, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:37:22.014900 22335 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 16:37:22.016117 22335 webserver.cc:460] Webserver started at http://127.0.98.62:41729/ using document root <none> and password file <none>
I20260925 16:37:22.016692 22335 fs_manager.cc:359] Metadata directory not provided
I20260925 16:37:22.016754 22335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:37:22.016918 22335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:37:22.019033 22335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790353966814071-392-0/minicluster-data/master-0/data/instance:
uuid: "900a044732184f65a4825b223eaadd83"
format_stamp: "Formatted at 2026-09-25 16:37:22 on dist-test-slave-65pf"
server_key: "106841746e7aca1caf158a23e9c57cf8"
server_key_iv: "21e356013da3e9f6c955c598814c28c2"
server_key_version: "encryptionkey@0"
I20260925 16:37:22.019534 22335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790353966814071-392-0/minicluster-data/master-0/wal/instance:
uuid: "900a044732184f65a4825b223eaadd83"
format_stamp: "Formatted at 2026-09-25 16:37:22 on dist-test-slave-65pf"
server_key: "106841746e7aca1caf158a23e9c57cf8"
server_key_iv: "21e356013da3e9f6c955c598814c28c2"
server_key_version: "encryptionkey@0"
I20260925 16:37:22.023078 22335 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 16:37:22.025341 22350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:37:22.026496 22335 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 16:37:22.026655 22335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790353966814071-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790353966814071-392-0/minicluster-data/master-0/wal
uuid: "900a044732184f65a4825b223eaadd83"
format_stamp: "Formatted at 2026-09-25 16:37:22 on dist-test-slave-65pf"
server_key: "106841746e7aca1caf158a23e9c57cf8"
server_key_iv: "21e356013da3e9f6c955c598814c28c2"
server_key_version: "encryptionkey@0"
I20260925 16:37:22.026795 22335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790353966814071-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790353966814071-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790353966814071-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:37:22.057051 22335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:37:22.061476 22335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:37:22.061748 22335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:37:22.070914 22335 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39379
I20260925 16:37:22.070960 22402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39379 every 8 connection(s)
I20260925 16:37:22.072116 22335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:37:22.075068 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 22335
I20260925 16:37:22.075222 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790353966814071-392-0/minicluster-data/master-0/wal/instance
I20260925 16:37:22.075489 392 external_mini_cluster.cc:1428] Setting key 3a426b5e4450e036853fa009c3ef56d2
I20260925 16:37:22.075986 22403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:37:22.085088 22403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 900a044732184f65a4825b223eaadd83: Bootstrap starting.
Sep 25 16:37:22 dist-test-slave-65pf krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354241, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:37:22.088312 22403 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 900a044732184f65a4825b223eaadd83: Neither blocks nor log segments found. Creating new log.
I20260925 16:37:22.089387 22403 log.cc:824] T 00000000000000000000000000000000 P 900a044732184f65a4825b223eaadd83: Log is configured to *not* fsync() on all Append() calls
I20260925 16:37:22.090368 22406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:22.077282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48373 (local address 127.0.98.62:39379)
0925 16:37:22.077659 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:22.077676 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:22.077738 (+ 62us) server_negotiation.cc:407] Connection header received
0925 16:37:22.078874 (+ 1136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:22.078908 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:22.079361 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:22.079686 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:22.080491 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:22.081560 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:22.082423 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:22.083108 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:22.083277 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:22.085780 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:37:22.085785 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:37:22.085797 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:37:22.085822 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:37:22.087520 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:22.087949 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:22.087959 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:22.087964 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:22.088066 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:22.088339 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:22.088344 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:22.088346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:22.088712 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:37:22.088905 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:22.089200 (+ 295us) server_negotiation.cc:299] Negotiation successful
0925 16:37:22.089514 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":126,"threads_started":1}
I20260925 16:37:22.091617 22403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 900a044732184f65a4825b223eaadd83: No bootstrap required, opened a new log
I20260925 16:37:22.093129 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:37:22.095620 22403 raft_consensus.cc:348] T 00000000000000000000000000000000 P 900a044732184f65a4825b223eaadd83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "900a044732184f65a4825b223eaadd83" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39379 } }
I20260925 16:37:22.095868 22403 raft_consensus.cc:374] T 00000000000000000000000000000000 P 900a044732184f65a4825b223eaadd83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:37:22.095940 22403 raft_consensus.cc:729] T 00000000000000000000000000000000 P 900a044732184f65a4825b223eaadd83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 900a044732184f65a4825b223eaadd83, State: Initialized, Role: FOLLOWER
I20260925 16:37:22.096369 22403 consensus_queue.cc:260] T 00000000000000000000000000000000 P 900a044732184f65a4825b223eaadd83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "900a044732184f65a4825b223eaadd83" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39379 } }
I20260925 16:37:22.096508 22403 raft_consensus.cc:388] T 00000000000000000000000000000000 P 900a044732184f65a4825b223eaadd83 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:37:22.096586 22403 raft_consensus.cc:482] T 00000000000000000000000000000000 P 900a044732184f65a4825b223eaadd83 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:37:22.096688 22403 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 900a044732184f65a4825b223eaadd83 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:37:22.097549 22403 raft_consensus.cc:504] T 00000000000000000000000000000000 P 900a044732184f65a4825b223eaadd83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "900a044732184f65a4825b223eaadd83" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39379 } }
I20260925 16:37:22.097898 22403 leader_election.cc:302] T 00000000000000000000000000000000 P 900a044732184f65a4825b223eaadd83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 900a044732184f65a4825b223eaadd83; no voters:
I20260925 16:37:22.098204 22403 leader_election.cc:288] T 00000000000000000000000000000000 P 900a044732184f65a4825b223eaadd83 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:37:22.098274 22413 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 900a044732184f65a4825b223eaadd83 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:37:22.098482 22413 raft_consensus.cc:686] T 00000000000000000000000000000000 P 900a044732184f65a4825b223eaadd83 [term 1 LEADER]: Becoming Leader. State: Replica: 900a044732184f65a4825b223eaadd83, State: Running, Role: LEADER
I20260925 16:37:22.098814 22413 consensus_queue.cc:237] T 00000000000000000000000000000000 P 900a044732184f65a4825b223eaadd83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "900a044732184f65a4825b223eaadd83" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39379 } }
I20260925 16:37:22.099501 22403 sys_catalog.cc:564] T 00000000000000000000000000000000 P 900a044732184f65a4825b223eaadd83 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:37:22.100512 22414 sys_catalog.cc:455] T 00000000000000000000000000000000 P 900a044732184f65a4825b223eaadd83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 900a044732184f65a4825b223eaadd83. Latest consensus state: current_term: 1 leader_uuid: "900a044732184f65a4825b223eaadd83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "900a044732184f65a4825b223eaadd83" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39379 } } }
I20260925 16:37:22.100634 22414 sys_catalog.cc:458] T 00000000000000000000000000000000 P 900a044732184f65a4825b223eaadd83 [sys.catalog]: This master's current role is: LEADER
I20260925 16:37:22.101008 22422 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:37:22.100993 22415 sys_catalog.cc:455] T 00000000000000000000000000000000 P 900a044732184f65a4825b223eaadd83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "900a044732184f65a4825b223eaadd83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "900a044732184f65a4825b223eaadd83" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39379 } } }
I20260925 16:37:22.101845 22415 sys_catalog.cc:458] T 00000000000000000000000000000000 P 900a044732184f65a4825b223eaadd83 [sys.catalog]: This master's current role is: LEADER
I20260925 16:37:22.104270 22422 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:37:22.104445 22406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:22.095127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44407 (local address 127.0.98.62:39379)
0925 16:37:22.095238 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:22.095241 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:22.095435 (+ 194us) server_negotiation.cc:407] Connection header received
0925 16:37:22.095571 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:22.095574 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:22.095614 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:22.095700 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:22.096412 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:22.097059 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:22.097887 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:22.098207 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:22.098315 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:22.099891 (+ 1576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:37:22.099897 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:37:22.099899 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:37:22.099926 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:37:22.102854 (+ 2928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:22.103252 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:22.103257 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:22.103260 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:22.103340 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:22.103590 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:22.103595 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:22.103598 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:22.103805 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:37:22.103922 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:22.104186 (+ 264us) server_negotiation.cc:299] Negotiation successful
0925 16:37:22.104293 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 16:37:22.110468 22422 catalog_manager.cc:1347] Generated new cluster ID: b029ce2d8aab45e9b4874b693e5b921b
I20260925 16:37:22.110539 22422 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:37:22.115062 22422 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:37:22.115869 22422 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:37:22.121378 22422 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 900a044732184f65a4825b223eaadd83: Generated new TSK 0
I20260925 16:37:22.121968 22422 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:37:22.131026 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 22335
2026-09-25T16:37:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1383 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790353966814071-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:37:22 dist-test-slave-65pf krb5kdc[22434](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:37:22 dist-test-slave-65pf krb5kdc[22434](info): set up 2 sockets
Sep 25 16:37:22 dist-test-slave-65pf krb5kdc[22434](info): commencing operation
krb5kdc: starting...
W20260925 16:37:23.182827 392 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790353966814071-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790353966814071-392-0/krb5kdc/test-user.keytab.
Sep 25 16:37:23 dist-test-slave-65pf krb5kdc[22434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:37:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:37:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 16:37:23.297609 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790353966814071-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790353966814071-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790353966814071-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45375
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45375
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:37:23.405390 22447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:37:23.405709 22447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:37:23.405781 22447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:37:23.409765 22447 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:37:23.409865 22447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:37:23.409902 22447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:37:23.409956 22447 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:37:23.409996 22447 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:37:23.415261 22447 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790353966814071-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790353966814071-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45375
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790353966814071-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:37:23.416365 22447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:37:23.417411 22447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:37:23.425693 22453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:37:23.425702 22452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:37:23.425705 22455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:37:23.425870 22447 server_base.cc:1034] running on GCE node
I20260925 16:37:23.426654 22447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:37:23.427747 22447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:37:23.428901 22447 hybrid_clock.cc:648] HybridClock initialized: now 1790354243428886 us; error 30 us; skew 500 ppm
Sep 25 16:37:23 dist-test-slave-65pf krb5kdc[22434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354243, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:37:23.431617 22447 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 16:37:23.432480 22447 webserver.cc:460] Webserver started at http://127.0.98.62:33731/ using document root <none> and password file <none>
I20260925 16:37:23.432986 22447 fs_manager.cc:359] Metadata directory not provided
I20260925 16:37:23.433041 22447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:37:23.433202 22447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:37:23.435225 22447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790353966814071-392-0/minicluster-data/master-0/data/instance:
uuid: "f8a0d8b4e41f4af7b6672094898f6b9a"
format_stamp: "Formatted at 2026-09-25 16:37:23 on dist-test-slave-65pf"
server_key: "302417a6a582b0db014de031a9a35bb4"
server_key_iv: "2f1bec389594b749dc78f522fd9be426"
server_key_version: "encryptionkey@0"
I20260925 16:37:23.435731 22447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790353966814071-392-0/minicluster-data/master-0/wal/instance:
uuid: "f8a0d8b4e41f4af7b6672094898f6b9a"
format_stamp: "Formatted at 2026-09-25 16:37:23 on dist-test-slave-65pf"
server_key: "302417a6a582b0db014de031a9a35bb4"
server_key_iv: "2f1bec389594b749dc78f522fd9be426"
server_key_version: "encryptionkey@0"
I20260925 16:37:23.439320 22447 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 16:37:23.441636 22462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:37:23.442950 22447 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260925 16:37:23.443084 22447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790353966814071-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790353966814071-392-0/minicluster-data/master-0/wal
uuid: "f8a0d8b4e41f4af7b6672094898f6b9a"
format_stamp: "Formatted at 2026-09-25 16:37:23 on dist-test-slave-65pf"
server_key: "302417a6a582b0db014de031a9a35bb4"
server_key_iv: "2f1bec389594b749dc78f522fd9be426"
server_key_version: "encryptionkey@0"
I20260925 16:37:23.443207 22447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790353966814071-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790353966814071-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790353966814071-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:37:23.463160 22447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:37:23.466606 22447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:37:23.466842 22447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:37:23.473907 22447 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45375
I20260925 16:37:23.473934 22514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45375 every 8 connection(s)
I20260925 16:37:23.474884 22447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:37:23.477660 22515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:37:23.483285 22515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8a0d8b4e41f4af7b6672094898f6b9a: Bootstrap starting.
I20260925 16:37:23.483449 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 22447
I20260925 16:37:23.483619 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790353966814071-392-0/minicluster-data/master-0/wal/instance
I20260925 16:37:23.483903 392 external_mini_cluster.cc:1428] Setting key 1a0e3d8c8fa89af12b67ca1b8389719e
I20260925 16:37:23.485968 22515 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8a0d8b4e41f4af7b6672094898f6b9a: Neither blocks nor log segments found. Creating new log.
I20260925 16:37:23.486769 22515 log.cc:824] T 00000000000000000000000000000000 P f8a0d8b4e41f4af7b6672094898f6b9a: Log is configured to *not* fsync() on all Append() calls
I20260925 16:37:23.488652 22515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8a0d8b4e41f4af7b6672094898f6b9a: No bootstrap required, opened a new log
I20260925 16:37:23.492398 22515 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8a0d8b4e41f4af7b6672094898f6b9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8a0d8b4e41f4af7b6672094898f6b9a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45375 } }
I20260925 16:37:23.492637 22515 raft_consensus.cc:374] T 00000000000000000000000000000000 P f8a0d8b4e41f4af7b6672094898f6b9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:37:23.492709 22515 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8a0d8b4e41f4af7b6672094898f6b9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8a0d8b4e41f4af7b6672094898f6b9a, State: Initialized, Role: FOLLOWER
I20260925 16:37:23.492807 22518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:23.485322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33995 (local address 127.0.98.62:45375)
0925 16:37:23.485747 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:23.485762 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:23.485797 (+ 35us) server_negotiation.cc:407] Connection header received
0925 16:37:23.486481 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:23.486499 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:23.486940 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:23.487286 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:23.487702 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:23.488647 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:23.489532 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:23.490002 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:23.490167 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:23.491039 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:37:23.491050 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:37:23.491064 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:37:23.491459 (+ 395us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:37:23.491512 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:37:23.491535 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:37:23.491641 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:23.491803 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 16:37:23.492012 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":137,"threads_started":1}
I20260925 16:37:23.493214 22515 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8a0d8b4e41f4af7b6672094898f6b9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8a0d8b4e41f4af7b6672094898f6b9a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45375 } }
I20260925 16:37:23.493362 22515 raft_consensus.cc:388] T 00000000000000000000000000000000 P f8a0d8b4e41f4af7b6672094898f6b9a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:37:23.493443 22515 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8a0d8b4e41f4af7b6672094898f6b9a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:37:23.493564 22515 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8a0d8b4e41f4af7b6672094898f6b9a [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:37:23.494693 22515 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8a0d8b4e41f4af7b6672094898f6b9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8a0d8b4e41f4af7b6672094898f6b9a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45375 } }
I20260925 16:37:23.495075 22515 leader_election.cc:302] T 00000000000000000000000000000000 P f8a0d8b4e41f4af7b6672094898f6b9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f8a0d8b4e41f4af7b6672094898f6b9a; no voters:
I20260925 16:37:23.495419 22515 leader_election.cc:288] T 00000000000000000000000000000000 P f8a0d8b4e41f4af7b6672094898f6b9a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:37:23.495502 22520 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f8a0d8b4e41f4af7b6672094898f6b9a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:37:23.495764 22520 raft_consensus.cc:686] T 00000000000000000000000000000000 P f8a0d8b4e41f4af7b6672094898f6b9a [term 1 LEADER]: Becoming Leader. State: Replica: f8a0d8b4e41f4af7b6672094898f6b9a, State: Running, Role: LEADER
I20260925 16:37:23.495849 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:37:23.496114 22520 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8a0d8b4e41f4af7b6672094898f6b9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8a0d8b4e41f4af7b6672094898f6b9a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45375 } }
I20260925 16:37:23.496580 22515 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8a0d8b4e41f4af7b6672094898f6b9a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:37:23.497753 22521 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8a0d8b4e41f4af7b6672094898f6b9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f8a0d8b4e41f4af7b6672094898f6b9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8a0d8b4e41f4af7b6672094898f6b9a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45375 } } }
I20260925 16:37:23.497891 22521 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8a0d8b4e41f4af7b6672094898f6b9a [sys.catalog]: This master's current role is: LEADER
I20260925 16:37:23.498138 22521 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8a0d8b4e41f4af7b6672094898f6b9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8a0d8b4e41f4af7b6672094898f6b9a. Latest consensus state: current_term: 1 leader_uuid: "f8a0d8b4e41f4af7b6672094898f6b9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8a0d8b4e41f4af7b6672094898f6b9a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45375 } } }
I20260925 16:37:23.498217 22521 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8a0d8b4e41f4af7b6672094898f6b9a [sys.catalog]: This master's current role is: LEADER
I20260925 16:37:23.498602 22534 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:37:23.501837 22534 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:37:23.503453 22518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:23.498256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49985 (local address 127.0.98.62:45375)
0925 16:37:23.498361 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:23.498365 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:23.499354 (+ 989us) server_negotiation.cc:407] Connection header received
0925 16:37:23.499466 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:23.499470 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:23.499516 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:23.499594 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:23.499839 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:23.500583 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:23.501695 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:23.502011 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:23.502130 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:23.503018 (+ 888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:37:23.503022 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:37:23.503025 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:37:23.503038 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:37:23.503068 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:37:23.503080 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:37:23.503157 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:23.503262 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 16:37:23.503314 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 16:37:23.508054 22534 catalog_manager.cc:1347] Generated new cluster ID: 207f31ad7eee4331bd52b99f8adc4139
I20260925 16:37:23.508134 22534 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:37:23.524966 22534 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:37:23.526003 22534 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:37:23.537773 22534 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f8a0d8b4e41f4af7b6672094898f6b9a: Generated new TSK 0
I20260925 16:37:23.538564 22534 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:37:23.545071 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 22447
2026-09-25T16:37:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1414 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790353966814071-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:37:23 dist-test-slave-65pf krb5kdc[22546](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:37:23 dist-test-slave-65pf krb5kdc[22546](info): set up 2 sockets
Sep 25 16:37:23 dist-test-slave-65pf krb5kdc[22546](info): commencing operation
krb5kdc: starting...
W20260925 16:37:24.571494 392 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790353966814071-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790353966814071-392-0/krb5kdc/test-user.keytab.
Sep 25 16:37:24 dist-test-slave-65pf krb5kdc[22546](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:37:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:37:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 16:37:24.693871 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790353966814071-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790353966814071-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790353966814071-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43977
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43977
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:37:24.807793 22559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:37:24.808115 22559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:37:24.808198 22559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:37:24.812208 22559 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:37:24.812315 22559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:37:24.812356 22559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:37:24.812415 22559 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:37:24.812453 22559 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:37:24.817740 22559 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790353966814071-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790353966814071-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43977
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22559
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790353966814071-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:37:24.818813 22559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:37:24.819811 22559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 16:37:24.828429 22559 server_base.cc:1034] running on GCE node
W20260925 16:37:24.828383 22567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:37:24.828406 22565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:37:24.828383 22564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:37:24.829075 22559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:37:24.830191 22559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:37:24.831390 22559 hybrid_clock.cc:648] HybridClock initialized: now 1790354244831377 us; error 39 us; skew 500 ppm
Sep 25 16:37:24 dist-test-slave-65pf krb5kdc[22546](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354244, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:37:24.834089 22559 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 16:37:24.835035 22559 webserver.cc:460] Webserver started at http://127.0.98.62:37399/ using document root <none> and password file <none>
I20260925 16:37:24.835583 22559 fs_manager.cc:359] Metadata directory not provided
I20260925 16:37:24.835675 22559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:37:24.835925 22559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:37:24.837889 22559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790353966814071-392-0/minicluster-data/master-0/data/instance:
uuid: "4a5851b6a1284b6899fcddffd69aaaef"
format_stamp: "Formatted at 2026-09-25 16:37:24 on dist-test-slave-65pf"
server_key: "d7aad95bc89e35d7ba82a2b7ddf8a094"
server_key_iv: "ed00afc281af375639cae1df3223b455"
server_key_version: "encryptionkey@0"
I20260925 16:37:24.838431 22559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790353966814071-392-0/minicluster-data/master-0/wal/instance:
uuid: "4a5851b6a1284b6899fcddffd69aaaef"
format_stamp: "Formatted at 2026-09-25 16:37:24 on dist-test-slave-65pf"
server_key: "d7aad95bc89e35d7ba82a2b7ddf8a094"
server_key_iv: "ed00afc281af375639cae1df3223b455"
server_key_version: "encryptionkey@0"
I20260925 16:37:24.841872 22559 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 16:37:24.844328 22574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:37:24.845489 22559 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260925 16:37:24.845623 22559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790353966814071-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790353966814071-392-0/minicluster-data/master-0/wal
uuid: "4a5851b6a1284b6899fcddffd69aaaef"
format_stamp: "Formatted at 2026-09-25 16:37:24 on dist-test-slave-65pf"
server_key: "d7aad95bc89e35d7ba82a2b7ddf8a094"
server_key_iv: "ed00afc281af375639cae1df3223b455"
server_key_version: "encryptionkey@0"
I20260925 16:37:24.845753 22559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790353966814071-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790353966814071-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790353966814071-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:37:24.850050 22559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:37:24.852752 22559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:37:24.852941 22559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:37:24.861654 22559 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43977
I20260925 16:37:24.861678 22626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43977 every 8 connection(s)
I20260925 16:37:24.862756 22559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:37:24.865535 22627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:37:24.870110 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 22559
I20260925 16:37:24.870270 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790353966814071-392-0/minicluster-data/master-0/wal/instance
I20260925 16:37:24.870554 392 external_mini_cluster.cc:1428] Setting key fd80f371e2b41ffd90a8889df7d28abe
I20260925 16:37:24.871167 22627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a5851b6a1284b6899fcddffd69aaaef: Bootstrap starting.
I20260925 16:37:24.873586 22627 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a5851b6a1284b6899fcddffd69aaaef: Neither blocks nor log segments found. Creating new log.
I20260925 16:37:24.874471 22627 log.cc:824] T 00000000000000000000000000000000 P 4a5851b6a1284b6899fcddffd69aaaef: Log is configured to *not* fsync() on all Append() calls
I20260925 16:37:24.875654 22630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:24.871886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45309 (local address 127.0.98.62:43977)
0925 16:37:24.872334 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:24.872348 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:24.872382 (+ 34us) server_negotiation.cc:407] Connection header received
0925 16:37:24.873034 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:24.873050 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:24.873331 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:24.873669 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:24.874438 (+ 769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:37:24.874443 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:37:24.874454 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:37:24.874464 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:37:24.874515 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:37:24.874533 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:37:24.874692 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:24.874803 (+ 111us) server_negotiation.cc:299] Negotiation successful
0925 16:37:24.874964 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":127,"threads_started":1}
I20260925 16:37:24.876511 22627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a5851b6a1284b6899fcddffd69aaaef: No bootstrap required, opened a new log
I20260925 16:37:24.878461 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:37:24.881230 22627 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a5851b6a1284b6899fcddffd69aaaef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a5851b6a1284b6899fcddffd69aaaef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43977 } }
I20260925 16:37:24.881556 22627 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a5851b6a1284b6899fcddffd69aaaef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:37:24.881623 22627 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a5851b6a1284b6899fcddffd69aaaef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a5851b6a1284b6899fcddffd69aaaef, State: Initialized, Role: FOLLOWER
I20260925 16:37:24.882159 22627 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a5851b6a1284b6899fcddffd69aaaef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a5851b6a1284b6899fcddffd69aaaef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43977 } }
I20260925 16:37:24.882330 22630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:24.880554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60301 (local address 127.0.98.62:43977)
0925 16:37:24.880694 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:24.880700 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:24.880883 (+ 183us) server_negotiation.cc:407] Connection header received
0925 16:37:24.881012 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:24.881014 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:24.881043 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:24.881108 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:24.881885 (+ 777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:37:24.881888 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:37:24.881890 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:37:24.881900 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:37:24.881924 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:37:24.881934 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:37:24.882003 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:24.882113 (+ 110us) server_negotiation.cc:299] Negotiation successful
0925 16:37:24.882142 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 16:37:24.882337 22627 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a5851b6a1284b6899fcddffd69aaaef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:37:24.882462 22627 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a5851b6a1284b6899fcddffd69aaaef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:37:24.882565 22627 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a5851b6a1284b6899fcddffd69aaaef [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:37:24.883625 22627 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a5851b6a1284b6899fcddffd69aaaef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a5851b6a1284b6899fcddffd69aaaef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43977 } }
I20260925 16:37:24.884016 22627 leader_election.cc:302] T 00000000000000000000000000000000 P 4a5851b6a1284b6899fcddffd69aaaef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a5851b6a1284b6899fcddffd69aaaef; no voters:
I20260925 16:37:24.884306 22627 leader_election.cc:288] T 00000000000000000000000000000000 P 4a5851b6a1284b6899fcddffd69aaaef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:37:24.884421 22637 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a5851b6a1284b6899fcddffd69aaaef [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:37:24.884652 22637 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a5851b6a1284b6899fcddffd69aaaef [term 1 LEADER]: Becoming Leader. State: Replica: 4a5851b6a1284b6899fcddffd69aaaef, State: Running, Role: LEADER
I20260925 16:37:24.885040 22637 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a5851b6a1284b6899fcddffd69aaaef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a5851b6a1284b6899fcddffd69aaaef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43977 } }
I20260925 16:37:24.885630 22627 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a5851b6a1284b6899fcddffd69aaaef [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:37:24.886826 22639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a5851b6a1284b6899fcddffd69aaaef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a5851b6a1284b6899fcddffd69aaaef. Latest consensus state: current_term: 1 leader_uuid: "4a5851b6a1284b6899fcddffd69aaaef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a5851b6a1284b6899fcddffd69aaaef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43977 } } }
I20260925 16:37:24.886831 22638 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a5851b6a1284b6899fcddffd69aaaef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a5851b6a1284b6899fcddffd69aaaef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a5851b6a1284b6899fcddffd69aaaef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43977 } } }
I20260925 16:37:24.886961 22638 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a5851b6a1284b6899fcddffd69aaaef [sys.catalog]: This master's current role is: LEADER
I20260925 16:37:24.886961 22639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a5851b6a1284b6899fcddffd69aaaef [sys.catalog]: This master's current role is: LEADER
I20260925 16:37:24.888360 22646 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:37:24.891283 22646 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:37:24.896363 22646 catalog_manager.cc:1347] Generated new cluster ID: e67ed4351d5c4693989972d8f483e569
I20260925 16:37:24.896440 22646 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:37:24.912810 22646 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:37:24.913175 22646 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:37:24.922200 22646 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a5851b6a1284b6899fcddffd69aaaef: Generated new TSK 0
I20260925 16:37:24.923089 22646 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:37:24.937361 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 22559
2026-09-25T16:37:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1392 ms)
[----------] 18 tests from AuthTokenIssuingTest (25098 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353966814071-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:37:24 dist-test-slave-65pf krb5kdc[22658](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:37:24 dist-test-slave-65pf krb5kdc[22658](info): set up 2 sockets
Sep 25 16:37:24 dist-test-slave-65pf krb5kdc[22658](info): commencing operation
krb5kdc: starting...
W20260925 16:37:25.962853 392 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353966814071-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353966814071-392-0/krb5kdc/test-user.keytab.
Sep 25 16:37:26 dist-test-slave-65pf krb5kdc[22658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:37:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:37:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 16:37:26.082389 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353966814071-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353966814071-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353966814071-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32819
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32819,127.0.98.61:46101,127.0.98.60:46117
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:37:26.187263 22671 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 16:37:26.187599 22671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:37:26.187674 22671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:37:26.187759 22671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:37:26.191694 22671 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:37:26.191789 22671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:37:26.191829 22671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:37:26.191861 22671 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:37:26.191887 22671 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:37:26.196929 22671 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353966814071-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353966814071-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32819,127.0.98.61:46101,127.0.98.60:46117
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353966814071-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:37:26.197968 22671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:37:26.198985 22671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:37:26.207237 22679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:37:26.207206 22677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:37:26.207257 22676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:37:26.207598 22671 server_base.cc:1034] running on GCE node
I20260925 16:37:26.208195 22671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:37:26.209378 22671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:37:26.210608 22671 hybrid_clock.cc:648] HybridClock initialized: now 1790354246210567 us; error 31 us; skew 500 ppm
Sep 25 16:37:26 dist-test-slave-65pf krb5kdc[22658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354246, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:37:26.213333 22671 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 16:37:26.214286 22671 webserver.cc:460] Webserver started at http://127.0.98.62:44401/ using document root <none> and password file <none>
I20260925 16:37:26.214918 22671 fs_manager.cc:359] Metadata directory not provided
I20260925 16:37:26.215013 22671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:37:26.215248 22671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:37:26.217334 22671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353966814071-392-0/minicluster-data/master-0/data/instance:
uuid: "e305eb7d1f1c4c7da776df775aacfd23"
format_stamp: "Formatted at 2026-09-25 16:37:26 on dist-test-slave-65pf"
server_key: "ac155e129fdd6561cd22e8e7c9da42e9"
server_key_iv: "30a83507bcbe4179515c69cc1a8bb846"
server_key_version: "encryptionkey@0"
I20260925 16:37:26.217896 22671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353966814071-392-0/minicluster-data/master-0/wal/instance:
uuid: "e305eb7d1f1c4c7da776df775aacfd23"
format_stamp: "Formatted at 2026-09-25 16:37:26 on dist-test-slave-65pf"
server_key: "ac155e129fdd6561cd22e8e7c9da42e9"
server_key_iv: "30a83507bcbe4179515c69cc1a8bb846"
server_key_version: "encryptionkey@0"
I20260925 16:37:26.221696 22671 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260925 16:37:26.224112 22686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:37:26.225569 22671 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 16:37:26.225740 22671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353966814071-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353966814071-392-0/minicluster-data/master-0/wal
uuid: "e305eb7d1f1c4c7da776df775aacfd23"
format_stamp: "Formatted at 2026-09-25 16:37:26 on dist-test-slave-65pf"
server_key: "ac155e129fdd6561cd22e8e7c9da42e9"
server_key_iv: "30a83507bcbe4179515c69cc1a8bb846"
server_key_version: "encryptionkey@0"
I20260925 16:37:26.225867 22671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353966814071-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353966814071-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353966814071-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:37:26.262066 22671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:37:26.265089 22671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:37:26.265321 22671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:37:26.272401 22671 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32819
I20260925 16:37:26.272681 22738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32819 every 8 connection(s)
I20260925 16:37:26.273406 22671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:37:26.276252 22739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:37:26.278403 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 22671
I20260925 16:37:26.278549 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353966814071-392-0/minicluster-data/master-0/wal/instance
I20260925 16:37:26.278887 392 external_mini_cluster.cc:1428] Setting key 863f7438b5f74f4be708c2cde3f068c3
I20260925 16:37:26.281103 22739 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32819 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 25 16:37:26 dist-test-slave-65pf krb5kdc[22658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354246, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:37:26.297143 22743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:26.283512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32819 (local address 127.0.0.1:37586)
0925 16:37:26.284110 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0925 16:37:26.284155 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 16:37:26.284997 (+ 842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:37:26.285752 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:37:26.285757 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:37:26.286212 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:37:26.286452 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:37:26.286466 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:37:26.287648 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:37:26.287652 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:37:26.288387 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:37:26.288396 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:37:26.289099 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:37:26.289104 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:37:26.289275 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:26.289868 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:37:26.289897 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:37:26.291904 (+ 2007us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:37:26.294008 (+ 2104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:37:26.294019 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:37:26.294028 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:37:26.294271 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:37:26.294661 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:37:26.294665 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:37:26.294667 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:37:26.294787 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:37:26.295409 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:37:26.295415 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:37:26.295564 (+ 149us) client_negotiation.cc:769] Sending connection context
0925 16:37:26.295753 (+ 189us) client_negotiation.cc:240] Negotiation successful
0925 16:37:26.296135 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"mutex_wait_us":128,"spinlock_wait_cycles":33664,"thread_start_us":175,"threads_started":1}
I20260925 16:37:26.297148 22742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:26.283511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37586 (local address 127.0.98.62:32819)
0925 16:37:26.283953 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:26.283973 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:26.284273 (+ 300us) server_negotiation.cc:407] Connection header received
0925 16:37:26.285327 (+ 1054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:26.285332 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:26.285534 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:26.285744 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:26.286630 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:26.287503 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:26.288537 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:26.288911 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:26.289206 (+ 295us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:26.292064 (+ 2858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:37:26.292071 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:37:26.292088 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:37:26.292109 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:37:26.293871 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:26.294410 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:26.294421 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:26.294422 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:26.294528 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:26.294891 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:26.294897 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:26.294900 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:26.295281 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:37:26.295440 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:26.295816 (+ 376us) server_negotiation.cc:299] Negotiation successful
0925 16:37:26.296072 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":293,"thread_start_us":149,"threads_started":1}
I20260925 16:37:26.299698 22739 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46101 } has no permanent_uuid. Determining permanent_uuid...
I20260925 16:37:26.300413 22743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:26.300165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46101 (local address 127.0.0.1:56508)
0925 16:37:26.300289 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 16:37:26.300357 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260925 16:37:26.300626 22689 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46101: connect: Connection refused (error 111)
I20260925 16:37:26.301501 22743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:26.301254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46101 (local address 127.0.0.1:56512)
0925 16:37:26.301390 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 16:37:26.301445 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 16:37:26.301671 22739 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46101: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46101: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353966814071-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353966814071-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260925 16:37:26.305583 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353966814071-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353966814071-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353966814071-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353966814071-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:46101
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353966814071-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32819,127.0.98.61:46101,127.0.98.60:46117
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 16:37:26.350898 22739 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46101 } attempt: 1
I20260925 16:37:26.351811 22743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:26.351507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46101 (local address 127.0.0.1:56526)
0925 16:37:26.351666 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 16:37:26.351735 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260925 16:37:26.352532 22743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:26.352315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46101 (local address 127.0.0.1:56538)
0925 16:37:26.352432 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 16:37:26.352479 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 16:37:26.352662 22739 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46101: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46101: connect: Connection refused (error 111)
W20260925 16:37:26.416088 22746 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 16:37:26.416409 22746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:37:26.416482 22746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:37:26.416555 22746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:37:26.420456 22746 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:37:26.420552 22746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:37:26.420590 22746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:37:26.420621 22746 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:37:26.420645 22746 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:37:26.420840 22739 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46101 } attempt: 2
I20260925 16:37:26.421657 22743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:26.421372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46101 (local address 127.0.0.1:56554)
0925 16:37:26.421523 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 16:37:26.421586 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260925 16:37:26.422323 22743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:26.422110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46101 (local address 127.0.0.1:56562)
0925 16:37:26.422214 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0925 16:37:26.422263 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 16:37:26.422443 22739 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46101: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46101: connect: Connection refused (error 111)
I20260925 16:37:26.426141 22746 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353966814071-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353966814071-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32819,127.0.98.61:46101,127.0.98.60:46117
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353966814071-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:46101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353966814071-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22746
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353966814071-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:37:26.427224 22746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:37:26.428194 22746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:37:26.436371 22751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:37:26.436414 22754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:37:26.436568 22746 server_base.cc:1034] running on GCE node
W20260925 16:37:26.436371 22752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:37:26.437157 22746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:37:26.438199 22746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:37:26.439388 22746 hybrid_clock.cc:648] HybridClock initialized: now 1790354246439371 us; error 29 us; skew 500 ppm
Sep 25 16:37:26 dist-test-slave-65pf krb5kdc[22658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354246, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:37:26.442114 22746 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260925 16:37:26.443064 22746 webserver.cc:460] Webserver started at http://127.0.98.61:38531/ using document root <none> and password file <none>
I20260925 16:37:26.443593 22746 fs_manager.cc:359] Metadata directory not provided
I20260925 16:37:26.443681 22746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:37:26.443902 22746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:37:26.445871 22746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353966814071-392-0/minicluster-data/master-1/data/instance:
uuid: "aa46560c36b64aa2afe1776d54d06cb3"
format_stamp: "Formatted at 2026-09-25 16:37:26 on dist-test-slave-65pf"
server_key: "4bc171ac7dc27fe18c75d69aab08982b"
server_key_iv: "be8434162b320a82816a6648400f0a51"
server_key_version: "encryptionkey@0"
I20260925 16:37:26.446430 22746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353966814071-392-0/minicluster-data/master-1/wal/instance:
uuid: "aa46560c36b64aa2afe1776d54d06cb3"
format_stamp: "Formatted at 2026-09-25 16:37:26 on dist-test-slave-65pf"
server_key: "4bc171ac7dc27fe18c75d69aab08982b"
server_key_iv: "be8434162b320a82816a6648400f0a51"
server_key_version: "encryptionkey@0"
I20260925 16:37:26.449977 22746 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 16:37:26.452337 22761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:37:26.453524 22746 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 16:37:26.453663 22746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353966814071-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353966814071-392-0/minicluster-data/master-1/wal
uuid: "aa46560c36b64aa2afe1776d54d06cb3"
format_stamp: "Formatted at 2026-09-25 16:37:26 on dist-test-slave-65pf"
server_key: "4bc171ac7dc27fe18c75d69aab08982b"
server_key_iv: "be8434162b320a82816a6648400f0a51"
server_key_version: "encryptionkey@0"
I20260925 16:37:26.453773 22746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353966814071-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353966814071-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353966814071-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:37:26.471210 22746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:37:26.474474 22746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:37:26.474706 22746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:37:26.481235 22746 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:46101
I20260925 16:37:26.481539 22813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:46101 every 8 connection(s)
I20260925 16:37:26.482159 22746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353966814071-392-0/minicluster-data/master-1/data/info.pb
I20260925 16:37:26.484866 22814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:37:26.488508 22814 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32819 } has no permanent_uuid. Determining permanent_uuid...
I20260925 16:37:26.491173 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 22746
I20260925 16:37:26.491292 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353966814071-392-0/minicluster-data/master-1/wal/instance
I20260925 16:37:26.491555 392 external_mini_cluster.cc:1428] Setting key 61eb5b8657e855cba65ffcb08122b201
Sep 25 16:37:26 dist-test-slave-65pf krb5kdc[22658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354246, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260925 16:37:26.501626 22742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:26.490327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37602 (local address 127.0.98.62:32819)
0925 16:37:26.490458 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:26.490467 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:26.491290 (+ 823us) server_negotiation.cc:407] Connection header received
0925 16:37:26.492260 (+ 970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:26.492263 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:26.492336 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:26.492437 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:26.493529 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:26.494078 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:26.495358 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:26.495603 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:26.495720 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:26.498098 (+ 2378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:37:26.498101 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:37:26.498103 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:37:26.498126 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:37:26.499547 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:26.500118 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:26.500123 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:26.500125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:26.500201 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:26.500532 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:26.500537 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:26.500539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:26.500709 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:37:26.500815 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:26.501335 (+ 520us) server_negotiation.cc:299] Negotiation successful
0925 16:37:26.501454 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 16:37:26.502442 22816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:26.490648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32819 (local address 127.0.0.1:37602)
0925 16:37:26.491128 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0925 16:37:26.491167 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 16:37:26.492054 (+ 887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:37:26.492597 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:37:26.492604 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:37:26.492939 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:37:26.493389 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:37:26.493401 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:37:26.494204 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:37:26.494207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:37:26.495237 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:37:26.495247 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:37:26.495763 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:37:26.495767 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:37:26.495926 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:26.496550 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:37:26.496564 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:37:26.497931 (+ 1367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:37:26.499665 (+ 1734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:37:26.499671 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:37:26.499684 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:37:26.500015 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:37:26.500300 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:37:26.500305 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:37:26.500307 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:37:26.500438 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:37:26.500822 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:37:26.500830 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:37:26.501075 (+ 245us) client_negotiation.cc:769] Sending connection context
0925 16:37:26.501284 (+ 209us) client_negotiation.cc:240] Negotiation successful
0925 16:37:26.501579 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":107,"threads_started":1}
I20260925 16:37:26.504163 22814 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46101 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 16:37:26 dist-test-slave-65pf krb5kdc[22658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354246, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 16:37:26.513592 22739 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46101 } attempt: 3
I20260925 16:37:26.514689 22816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:26.504680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46101 (local address 127.0.0.1:56578)
0925 16:37:26.504795 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0925 16:37:26.504810 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:37:26.504880 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:37:26.505464 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:37:26.505467 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:37:26.505631 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:37:26.505702 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:37:26.505707 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:37:26.506734 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:37:26.506737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:37:26.507228 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:37:26.507235 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:37:26.507832 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:37:26.507835 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:37:26.507940 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:26.508427 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:37:26.508437 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:37:26.511105 (+ 2668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:37:26.512984 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:37:26.512986 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:37:26.512988 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:37:26.513201 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:37:26.513466 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:37:26.513468 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:37:26.513470 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:37:26.513558 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:37:26.514097 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:37:26.514100 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:37:26.514207 (+ 107us) client_negotiation.cc:769] Sending connection context
0925 16:37:26.514340 (+ 133us) client_negotiation.cc:240] Negotiation successful
0925 16:37:26.514468 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 16:37:26.514873 22819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:26.504830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56578 (local address 127.0.98.61:46101)
0925 16:37:26.505129 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:26.505139 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:26.505156 (+ 17us) server_negotiation.cc:407] Connection header received
0925 16:37:26.505186 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:26.505190 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:26.505337 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:26.505490 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:26.505830 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:26.506545 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:26.507337 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:26.507707 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:26.507835 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:26.511265 (+ 3430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:37:26.511274 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:37:26.511284 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:37:26.511303 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:37:26.512886 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:26.513311 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:26.513316 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:26.513317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:26.513388 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:26.513651 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:26.513654 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:26.513655 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:26.513966 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:37:26.514118 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:26.514439 (+ 321us) server_negotiation.cc:299] Negotiation successful
0925 16:37:26.514640 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":138,"threads_started":1}
I20260925 16:37:26.516916 22814 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46117 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353966814071-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353966814071-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260925 16:37:26.517699 22816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:26.517420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46117 (local address 127.0.0.1:56016)
0925 16:37:26.517547 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 16:37:26.517630 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 16:37:26.518029 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353966814071-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353966814071-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353966814071-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353966814071-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:46117
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353966814071-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32819,127.0.98.61:46101,127.0.98.60:46117
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:37:26.518302 22762 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46117: connect: Connection refused (error 111)
I20260925 16:37:26.519613 22816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:26.519403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46117 (local address 127.0.0.1:56022)
0925 16:37:26.519511 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0925 16:37:26.519554 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 16:37:26.519809 22814 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46117: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46117: connect: Connection refused (error 111)
Sep 25 16:37:26 dist-test-slave-65pf krb5kdc[22658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354246, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 16:37:26.523993 22820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:26.514360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56594 (local address 127.0.98.61:46101)
0925 16:37:26.514742 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:26.514753 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:26.514767 (+ 14us) server_negotiation.cc:407] Connection header received
0925 16:37:26.514797 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:26.514800 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:26.514838 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:26.515326 (+ 488us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:26.515443 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:26.516169 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:26.517867 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:26.518284 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:26.518395 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:26.520823 (+ 2428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:37:26.520826 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:37:26.520828 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:37:26.520846 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:37:26.522347 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:26.522879 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:26.522881 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:26.522882 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:26.522950 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:26.523248 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:26.523250 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:26.523251 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:26.523387 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:37:26.523453 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:26.523789 (+ 336us) server_negotiation.cc:299] Negotiation successful
0925 16:37:26.523873 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":112,"threads_started":1}
I20260925 16:37:26.524125 22743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:26.514110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46101 (local address 127.0.0.1:56594)
0925 16:37:26.514246 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 16:37:26.514282 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 16:37:26.514361 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:37:26.514984 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:37:26.514987 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:37:26.515130 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:37:26.515237 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:37:26.515243 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:37:26.516298 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:37:26.516300 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:37:26.517706 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:37:26.517716 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:37:26.518425 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:37:26.518432 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:37:26.518577 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:26.519102 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:37:26.519122 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:37:26.520663 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:37:26.522457 (+ 1794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:37:26.522463 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:37:26.522468 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:37:26.522780 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:37:26.523039 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:37:26.523046 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:37:26.523050 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:37:26.523151 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:37:26.523476 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:37:26.523482 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:37:26.523603 (+ 121us) client_negotiation.cc:769] Sending connection context
0925 16:37:26.523735 (+ 132us) client_negotiation.cc:240] Negotiation successful
0925 16:37:26.523894 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 16:37:26.525071 22739 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46117 } has no permanent_uuid. Determining permanent_uuid...
I20260925 16:37:26.525832 22743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:26.525544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46117 (local address 127.0.0.1:56036)
0925 16:37:26.525684 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 16:37:26.525751 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 16:37:26.526671 22743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:26.526324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46117 (local address 127.0.0.1:56042)
0925 16:37:26.526495 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 16:37:26.526556 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260925 16:37:26.526800 22739 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46117: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46117: connect: Connection refused (error 111)
I20260925 16:37:26.557973 22739 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46117 } attempt: 1
I20260925 16:37:26.558842 22743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:26.558506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46117 (local address 127.0.0.1:56048)
0925 16:37:26.558686 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 16:37:26.558752 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260925 16:37:26.559535 22743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:26.559330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46117 (local address 127.0.0.1:56054)
0925 16:37:26.559433 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0925 16:37:26.559480 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260925 16:37:26.559664 22739 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46117: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46117: connect: Connection refused (error 111)
I20260925 16:37:26.569080 22814 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46117 } attempt: 1
I20260925 16:37:26.570001 22816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:26.569715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46117 (local address 127.0.0.1:56058)
0925 16:37:26.569862 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 16:37:26.569924 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260925 16:37:26.570797 22816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:26.570558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46117 (local address 127.0.0.1:56066)
0925 16:37:26.570693 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 16:37:26.570745 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260925 16:37:26.570940 22814 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46117: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46117: connect: Connection refused (error 111)
W20260925 16:37:26.631501 22822 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 16:37:26.631826 22822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:37:26.631894 22822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:37:26.631968 22822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 16:37:26.634805 22739 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46117 } attempt: 2
I20260925 16:37:26.635591 22743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:26.635281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46117 (local address 127.0.0.1:56072)
0925 16:37:26.635436 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 16:37:26.635502 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260925 16:37:26.636286 22743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:26.636081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46117 (local address 127.0.0.1:56076)
0925 16:37:26.636177 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0925 16:37:26.636230 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 16:37:26.636435 22739 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46117: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46117: connect: Connection refused (error 111)
W20260925 16:37:26.636758 22822 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:37:26.636804 22822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:37:26.636844 22822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:37:26.636883 22822 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:37:26.636931 22822 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:37:26.639094 22814 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46117 } attempt: 2
I20260925 16:37:26.639767 22816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:26.639486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46117 (local address 127.0.0.1:56084)
0925 16:37:26.639621 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 16:37:26.639704 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260925 16:37:26.640450 22816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:26.640245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46117 (local address 127.0.0.1:56094)
0925 16:37:26.640351 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0925 16:37:26.640397 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 16:37:26.640584 22814 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46117: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46117: connect: Connection refused (error 111)
I20260925 16:37:26.642490 22822 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353966814071-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353966814071-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32819,127.0.98.61:46101,127.0.98.60:46117
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353966814071-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:46117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353966814071-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353966814071-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:37:26.643589 22822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:37:26.644663 22822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:37:26.652966 22827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:37:26.652998 22830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:37:26.652966 22828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:37:26.653065 22822 server_base.cc:1034] running on GCE node
I20260925 16:37:26.653862 22822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:37:26.654947 22822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:37:26.656113 22822 hybrid_clock.cc:648] HybridClock initialized: now 1790354246656086 us; error 38 us; skew 500 ppm
Sep 25 16:37:26 dist-test-slave-65pf krb5kdc[22658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354246, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:37:26.659112 22822 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260925 16:37:26.659996 22822 webserver.cc:460] Webserver started at http://127.0.98.60:34365/ using document root <none> and password file <none>
I20260925 16:37:26.660569 22822 fs_manager.cc:359] Metadata directory not provided
I20260925 16:37:26.660663 22822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:37:26.660898 22822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:37:26.662967 22822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353966814071-392-0/minicluster-data/master-2/data/instance:
uuid: "6a9095c1d0f943648d6e9d91a1fc34fe"
format_stamp: "Formatted at 2026-09-25 16:37:26 on dist-test-slave-65pf"
server_key: "a87550b32d69d8d136ba1f8b1fd15c5d"
server_key_iv: "c8be29130081eef7d55b41337a428422"
server_key_version: "encryptionkey@0"
I20260925 16:37:26.663513 22822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353966814071-392-0/minicluster-data/master-2/wal/instance:
uuid: "6a9095c1d0f943648d6e9d91a1fc34fe"
format_stamp: "Formatted at 2026-09-25 16:37:26 on dist-test-slave-65pf"
server_key: "a87550b32d69d8d136ba1f8b1fd15c5d"
server_key_iv: "c8be29130081eef7d55b41337a428422"
server_key_version: "encryptionkey@0"
I20260925 16:37:26.667114 22822 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 16:37:26.669381 22837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:37:26.670559 22822 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:37:26.670754 22822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353966814071-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353966814071-392-0/minicluster-data/master-2/wal
uuid: "6a9095c1d0f943648d6e9d91a1fc34fe"
format_stamp: "Formatted at 2026-09-25 16:37:26 on dist-test-slave-65pf"
server_key: "a87550b32d69d8d136ba1f8b1fd15c5d"
server_key_iv: "c8be29130081eef7d55b41337a428422"
server_key_version: "encryptionkey@0"
I20260925 16:37:26.670866 22822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353966814071-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353966814071-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353966814071-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:37:26.694154 22822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:37:26.697497 22822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:37:26.697718 22822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:37:26.704427 22822 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:46117
I20260925 16:37:26.704758 22889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:46117 every 8 connection(s)
I20260925 16:37:26.705386 22822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353966814071-392-0/minicluster-data/master-2/data/info.pb
I20260925 16:37:26.708106 22890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:37:26.711717 22890 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32819 } has no permanent_uuid. Determining permanent_uuid...
I20260925 16:37:26.715080 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 22822
I20260925 16:37:26.715200 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353966814071-392-0/minicluster-data/master-2/wal/instance
I20260925 16:37:26.715487 392 external_mini_cluster.cc:1428] Setting key 825f7a990743f2fb1c9035a135fb7677
Sep 25 16:37:26 dist-test-slave-65pf krb5kdc[22658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354246, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 25 16:37:26 dist-test-slave-65pf krb5kdc[22658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 16:37:26.725944 22742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:26.713643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37614 (local address 127.0.98.62:32819)
0925 16:37:26.713828 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:26.713833 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:26.714665 (+ 832us) server_negotiation.cc:407] Connection header received
0925 16:37:26.715992 (+ 1327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:26.715997 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:26.716054 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:26.716139 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:26.717603 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:26.718110 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:26.719165 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:26.719376 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:26.719487 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:26.722001 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:37:26.722004 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:37:26.722006 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:37:26.722029 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:37:26.723735 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:26.724302 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:26.724308 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:26.724313 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:26.724401 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:26.724772 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:26.724775 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:26.724777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:26.725007 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:37:26.725075 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:26.725641 (+ 566us) server_negotiation.cc:299] Negotiation successful
0925 16:37:26.725781 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260925 16:37:26.726873 22892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:26.713972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32819 (local address 127.0.0.1:37614)
0925 16:37:26.714469 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0925 16:37:26.714513 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 16:37:26.715752 (+ 1239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:37:26.716371 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:37:26.716380 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:37:26.716844 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:37:26.717452 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:37:26.717467 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:37:26.718232 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:37:26.718236 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:37:26.719029 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:37:26.719038 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:37:26.719521 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:37:26.719525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:37:26.719701 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:26.720296 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:37:26.720318 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:37:26.721834 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:37:26.723840 (+ 2006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:37:26.723847 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:37:26.723855 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:37:26.724180 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:37:26.724510 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:37:26.724513 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:37:26.724515 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:37:26.724681 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:37:26.725108 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:37:26.725117 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:37:26.725359 (+ 242us) client_negotiation.cc:769] Sending connection context
0925 16:37:26.725574 (+ 215us) client_negotiation.cc:240] Negotiation successful
0925 16:37:26.725896 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":31,"thread_start_us":121,"threads_started":1}
I20260925 16:37:26.726877 22894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:26.716186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56106 (local address 127.0.98.60:46117)
0925 16:37:26.716553 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:26.716564 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:26.716588 (+ 24us) server_negotiation.cc:407] Connection header received
0925 16:37:26.716653 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:26.716658 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:26.716859 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:26.717020 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:26.717702 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:26.718541 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:26.719455 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:26.719810 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:26.719920 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:26.722477 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:37:26.722484 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:37:26.722496 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:37:26.722522 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:37:26.724268 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:26.724775 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:26.724780 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:26.724782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:26.724853 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:26.725133 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:26.725136 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:26.725138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:26.725368 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:37:26.725591 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:26.725824 (+ 233us) server_negotiation.cc:299] Negotiation successful
0925 16:37:26.725949 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":9,"server-negotiator.queue_time_us":275,"thread_start_us":104,"threads_started":1}
I20260925 16:37:26.728389 22890 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46101 } has no permanent_uuid. Determining permanent_uuid...
I20260925 16:37:26.731750 22814 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46117 } attempt: 3
Sep 25 16:37:26 dist-test-slave-65pf krb5kdc[22658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354246, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 25 16:37:26 dist-test-slave-65pf krb5kdc[22658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:37:26.735630 22739 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46117 } attempt: 3
I20260925 16:37:26.738173 22892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:26.728878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46101 (local address 127.0.0.1:56610)
0925 16:37:26.728974 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0925 16:37:26.728988 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:37:26.729058 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:37:26.729552 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:37:26.729556 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:37:26.729788 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:37:26.729896 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:37:26.729902 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:37:26.730936 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:37:26.730940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:37:26.731714 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:37:26.731723 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:37:26.732406 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:37:26.732409 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:37:26.732491 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:26.732959 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:37:26.732973 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:37:26.734531 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:37:26.736517 (+ 1986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:37:26.736521 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:37:26.736525 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:37:26.736757 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:37:26.737116 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:37:26.737122 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:37:26.737125 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:37:26.737205 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:37:26.737679 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:37:26.737681 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:37:26.737774 (+ 93us) client_negotiation.cc:769] Sending connection context
0925 16:37:26.737881 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 16:37:26.737992 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260925 16:37:26.738206 22820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:26.729065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56610 (local address 127.0.98.61:46101)
0925 16:37:26.729222 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:26.729226 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:26.729241 (+ 15us) server_negotiation.cc:407] Connection header received
0925 16:37:26.729284 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:26.729287 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:26.729338 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:26.729419 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:26.730028 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:26.730831 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:26.731831 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:26.732182 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:26.732283 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:26.734694 (+ 2411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:37:26.734698 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:37:26.734700 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:37:26.734726 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:37:26.736404 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:26.736892 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:26.736897 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:26.736902 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:26.737001 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:26.737326 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:26.737331 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:26.737333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:26.737540 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:37:26.737608 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:26.737950 (+ 342us) server_negotiation.cc:299] Negotiation successful
0925 16:37:26.738060 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Sep 25 16:37:26 dist-test-slave-65pf krb5kdc[22658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354246, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 16:37:26.740206 22890 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46117 } has no permanent_uuid. Determining permanent_uuid...
I20260925 16:37:26.741189 22742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:26.729456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37620 (local address 127.0.98.62:32819)
0925 16:37:26.730433 (+ 977us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:26.730436 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:26.730449 (+ 13us) server_negotiation.cc:407] Connection header received
0925 16:37:26.730479 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:26.730482 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:26.730516 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:26.730613 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:26.731258 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:26.731959 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:26.732947 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:26.733165 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:26.733325 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:26.735854 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:37:26.735860 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:37:26.735866 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:37:26.735895 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:37:26.738970 (+ 3075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:26.739459 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:26.739464 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:26.739466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:26.739549 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:26.739841 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:26.739844 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:26.739846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:26.740054 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:37:26.740302 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:26.740928 (+ 626us) server_negotiation.cc:299] Negotiation successful
0925 16:37:26.741058 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":907}
Sep 25 16:37:26 dist-test-slave-65pf krb5kdc[22658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354246, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 16:37:26.743700 22816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:26.732112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46117 (local address 127.0.0.1:56112)
0925 16:37:26.732240 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 16:37:26.732254 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:37:26.732335 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:37:26.732954 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:37:26.732957 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:37:26.733139 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:37:26.733244 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:37:26.733251 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:37:26.734187 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:37:26.734193 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:37:26.734837 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:37:26.734847 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:37:26.736028 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:37:26.736033 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:37:26.736386 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:26.737006 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:37:26.737020 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:37:26.738737 (+ 1717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:37:26.742115 (+ 3378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:37:26.742118 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:37:26.742120 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:37:26.742384 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:37:26.742705 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:37:26.742708 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:37:26.742709 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:37:26.742789 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:37:26.743212 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:37:26.743215 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:37:26.743299 (+ 84us) client_negotiation.cc:769] Sending connection context
0925 16:37:26.743425 (+ 126us) client_negotiation.cc:240] Negotiation successful
0925 16:37:26.743569 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260925 16:37:26.744326 22894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:26.732344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56112 (local address 127.0.98.60:46117)
0925 16:37:26.732469 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:26.732473 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:26.732483 (+ 10us) server_negotiation.cc:407] Connection header received
0925 16:37:26.732511 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:26.732513 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:26.732545 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:26.732612 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:26.733378 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:26.734073 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:26.735239 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:26.735894 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:26.737503 (+ 1609us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:26.738871 (+ 1368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:37:26.738875 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:37:26.738877 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:37:26.738901 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:37:26.742007 (+ 3106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:26.742494 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:26.742498 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:26.742499 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:26.742577 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:26.742899 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:26.742903 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:26.742906 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:26.743102 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:37:26.743229 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:26.744019 (+ 790us) server_negotiation.cc:299] Negotiation successful
0925 16:37:26.744177 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":60}
I20260925 16:37:26.745956 22743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:26.735948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46117 (local address 127.0.0.1:56122)
0925 16:37:26.736588 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0925 16:37:26.736602 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:37:26.736668 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:37:26.737399 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:37:26.737405 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:37:26.737574 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:37:26.737677 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:37:26.737683 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:37:26.738587 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:37:26.738622 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:37:26.739416 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:37:26.739430 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:37:26.740362 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:37:26.740366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:37:26.740449 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:26.740952 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:37:26.740970 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:37:26.742392 (+ 1422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:37:26.744374 (+ 1982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:37:26.744381 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:37:26.744383 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:37:26.744615 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:37:26.744920 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:37:26.744923 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:37:26.744924 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:37:26.745006 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:37:26.745502 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:37:26.745505 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:37:26.745592 (+ 87us) client_negotiation.cc:769] Sending connection context
0925 16:37:26.745669 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 16:37:26.745797 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562}
I20260925 16:37:26.746776 22895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:26.736802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56122 (local address 127.0.98.60:46117)
0925 16:37:26.737160 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:26.737168 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:26.737183 (+ 15us) server_negotiation.cc:407] Connection header received
0925 16:37:26.737227 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:26.737230 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:26.737268 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:26.737456 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:26.737787 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:26.738463 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:26.739780 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:26.740220 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:26.740659 (+ 439us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:26.742520 (+ 1861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:37:26.742523 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:37:26.742525 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:37:26.742551 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:37:26.744257 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:26.744730 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:26.744735 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:26.744738 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:26.744819 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:26.745158 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:26.745161 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:26.745162 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:26.745368 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:37:26.746384 (+ 1016us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:26.746455 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 16:37:26.746562 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":287,"thread_start_us":116,"threads_started":1}
I20260925 16:37:26.748078 22814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa46560c36b64aa2afe1776d54d06cb3: Bootstrap starting.
Sep 25 16:37:26 dist-test-slave-65pf krb5kdc[22658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 16:37:26.749621 22739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e305eb7d1f1c4c7da776df775aacfd23: Bootstrap starting.
I20260925 16:37:26.750967 22814 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa46560c36b64aa2afe1776d54d06cb3: Neither blocks nor log segments found. Creating new log.
I20260925 16:37:26.751801 22739 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e305eb7d1f1c4c7da776df775aacfd23: Neither blocks nor log segments found. Creating new log.
I20260925 16:37:26.751895 22814 log.cc:824] T 00000000000000000000000000000000 P aa46560c36b64aa2afe1776d54d06cb3: Log is configured to *not* fsync() on all Append() calls
I20260925 16:37:26.752421 22820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:26.742381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56624 (local address 127.0.98.61:46101)
0925 16:37:26.742526 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:26.742532 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:26.742544 (+ 12us) server_negotiation.cc:407] Connection header received
0925 16:37:26.742575 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:26.742577 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:26.742639 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:26.742711 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:26.743442 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:26.744185 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:26.745271 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:26.745623 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:26.745694 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:26.748872 (+ 3178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:37:26.748876 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:37:26.748878 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:37:26.748901 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:37:26.750462 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:26.750950 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:26.750953 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:26.750955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:26.751058 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:26.751412 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:26.751415 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:26.751417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:26.751606 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:37:26.751764 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:26.752123 (+ 359us) server_negotiation.cc:299] Negotiation successful
0925 16:37:26.752254 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":85}
I20260925 16:37:26.752604 22739 log.cc:824] T 00000000000000000000000000000000 P e305eb7d1f1c4c7da776df775aacfd23: Log is configured to *not* fsync() on all Append() calls
I20260925 16:37:26.754179 22814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa46560c36b64aa2afe1776d54d06cb3: No bootstrap required, opened a new log
Sep 25 16:37:26 dist-test-slave-65pf krb5kdc[22658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354246, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 16:37:26.755406 22739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e305eb7d1f1c4c7da776df775aacfd23: No bootstrap required, opened a new log
I20260925 16:37:26.755973 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:37:26.757579 22713 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "e305eb7d1f1c4c7da776df775aacfd23"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37620
I20260925 16:37:26.758186 22814 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa46560c36b64aa2afe1776d54d06cb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e305eb7d1f1c4c7da776df775aacfd23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32819 } } peers { permanent_uuid: "aa46560c36b64aa2afe1776d54d06cb3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46101 } } peers { permanent_uuid: "6a9095c1d0f943648d6e9d91a1fc34fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46117 } }
I20260925 16:37:26.758404 22814 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa46560c36b64aa2afe1776d54d06cb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa46560c36b64aa2afe1776d54d06cb3, State: Initialized, Role: FOLLOWER
I20260925 16:37:26.758644 22892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:26.741182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46117 (local address 127.0.0.1:56134)
0925 16:37:26.741586 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0925 16:37:26.741598 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 16:37:26.741663 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:37:26.744255 (+ 2592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:37:26.744260 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:37:26.744413 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:37:26.744484 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:37:26.744490 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:37:26.745467 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:37:26.745469 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:37:26.746234 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:37:26.746245 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:37:26.747008 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:37:26.747012 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:37:26.747107 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:26.753273 (+ 6166us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:37:26.753293 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:37:26.754701 (+ 1408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:37:26.757161 (+ 2460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:37:26.757164 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:37:26.757166 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:37:26.757393 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:37:26.757710 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:37:26.757713 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:37:26.757715 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:37:26.757791 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:37:26.758184 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:37:26.758187 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:37:26.758268 (+ 81us) client_negotiation.cc:769] Sending connection context
0925 16:37:26.758327 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 16:37:26.758457 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338}
I20260925 16:37:26.758912 22814 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa46560c36b64aa2afe1776d54d06cb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e305eb7d1f1c4c7da776df775aacfd23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32819 } } peers { permanent_uuid: "aa46560c36b64aa2afe1776d54d06cb3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46101 } } peers { permanent_uuid: "6a9095c1d0f943648d6e9d91a1fc34fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46117 } }
I20260925 16:37:26.759153 22896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:26.741366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56134 (local address 127.0.98.60:46117)
0925 16:37:26.743376 (+ 2010us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:26.743387 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:26.743400 (+ 13us) server_negotiation.cc:407] Connection header received
0925 16:37:26.743433 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:26.743436 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:26.743501 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:26.744145 (+ 644us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:26.744590 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:26.745316 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:26.746361 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:26.746793 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:26.746903 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:26.755166 (+ 8263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:37:26.755169 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:37:26.755171 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:37:26.755194 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:37:26.757022 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:26.757531 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:26.757533 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:26.757535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:26.757606 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:26.757875 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:26.757880 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:26.757883 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:26.758042 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:37:26.758820 (+ 778us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:26.758885 (+ 65us) server_negotiation.cc:299] Negotiation successful
0925 16:37:26.758992 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1616,"thread_start_us":106,"threads_started":1}
I20260925 16:37:26.760237 22739 raft_consensus.cc:348] T 00000000000000000000000000000000 P e305eb7d1f1c4c7da776df775aacfd23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e305eb7d1f1c4c7da776df775aacfd23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32819 } } peers { permanent_uuid: "aa46560c36b64aa2afe1776d54d06cb3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46101 } } peers { permanent_uuid: "6a9095c1d0f943648d6e9d91a1fc34fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46117 } }
I20260925 16:37:26.760424 22739 raft_consensus.cc:729] T 00000000000000000000000000000000 P e305eb7d1f1c4c7da776df775aacfd23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e305eb7d1f1c4c7da776df775aacfd23, State: Initialized, Role: FOLLOWER
I20260925 16:37:26.760525 22814 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa46560c36b64aa2afe1776d54d06cb3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:37:26.760717 22739 consensus_queue.cc:260] T 00000000000000000000000000000000 P e305eb7d1f1c4c7da776df775aacfd23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e305eb7d1f1c4c7da776df775aacfd23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32819 } } peers { permanent_uuid: "aa46560c36b64aa2afe1776d54d06cb3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46101 } } peers { permanent_uuid: "6a9095c1d0f943648d6e9d91a1fc34fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46117 } }
I20260925 16:37:26.761781 22890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a9095c1d0f943648d6e9d91a1fc34fe: Bootstrap starting.
I20260925 16:37:26.762007 22739 sys_catalog.cc:564] T 00000000000000000000000000000000 P e305eb7d1f1c4c7da776df775aacfd23 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:37:26.762105 22713 raft_consensus.cc:482] T 00000000000000000000000000000000 P e305eb7d1f1c4c7da776df775aacfd23 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 16:37:26.762228 22713 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e305eb7d1f1c4c7da776df775aacfd23 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:37:26.763538 22713 raft_consensus.cc:504] T 00000000000000000000000000000000 P e305eb7d1f1c4c7da776df775aacfd23 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e305eb7d1f1c4c7da776df775aacfd23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32819 } } peers { permanent_uuid: "aa46560c36b64aa2afe1776d54d06cb3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46101 } } peers { permanent_uuid: "6a9095c1d0f943648d6e9d91a1fc34fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46117 } }
I20260925 16:37:26.764287 22890 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a9095c1d0f943648d6e9d91a1fc34fe: Neither blocks nor log segments found. Creating new log.
I20260925 16:37:26.764420 22713 leader_election.cc:288] T 00000000000000000000000000000000 P e305eb7d1f1c4c7da776df775aacfd23 [CANDIDATE]: Term 1 election: Requested vote from peers aa46560c36b64aa2afe1776d54d06cb3 (127.0.98.61:46101), 6a9095c1d0f943648d6e9d91a1fc34fe (127.0.98.60:46117)
I20260925 16:37:26.765090 22890 log.cc:824] T 00000000000000000000000000000000 P 6a9095c1d0f943648d6e9d91a1fc34fe: Log is configured to *not* fsync() on all Append() calls
I20260925 16:37:26.765297 22900 sys_catalog.cc:455] T 00000000000000000000000000000000 P e305eb7d1f1c4c7da776df775aacfd23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e305eb7d1f1c4c7da776df775aacfd23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32819 } } peers { permanent_uuid: "aa46560c36b64aa2afe1776d54d06cb3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46101 } } peers { permanent_uuid: "6a9095c1d0f943648d6e9d91a1fc34fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46117 } } }
I20260925 16:37:26.765439 22900 sys_catalog.cc:458] T 00000000000000000000000000000000 P e305eb7d1f1c4c7da776df775aacfd23 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:37:26.767257 22890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a9095c1d0f943648d6e9d91a1fc34fe: No bootstrap required, opened a new log
I20260925 16:37:26.767421 22788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e305eb7d1f1c4c7da776df775aacfd23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "aa46560c36b64aa2afe1776d54d06cb3"
I20260925 16:37:26.767625 22788 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa46560c36b64aa2afe1776d54d06cb3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:37:26.764051 22746 master_runner.cc:186] Error getting master registration for 127.0.98.62:32819: OK, instance_id { permanent_uuid: "e305eb7d1f1c4c7da776df775aacfd23" instance_seqno: 1790354246270420 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 16:37:26.768726 22746 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 16:37:26.768853 22788 raft_consensus.cc:2454] T 00000000000000000000000000000000 P aa46560c36b64aa2afe1776d54d06cb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e305eb7d1f1c4c7da776df775aacfd23 in term 1.
I20260925 16:37:26.769223 22689 leader_election.cc:302] T 00000000000000000000000000000000 P e305eb7d1f1c4c7da776df775aacfd23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa46560c36b64aa2afe1776d54d06cb3, e305eb7d1f1c4c7da776df775aacfd23; no voters:
I20260925 16:37:26.769451 22900 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e305eb7d1f1c4c7da776df775aacfd23 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:37:26.769636 22900 raft_consensus.cc:686] T 00000000000000000000000000000000 P e305eb7d1f1c4c7da776df775aacfd23 [term 1 LEADER]: Becoming Leader. State: Replica: e305eb7d1f1c4c7da776df775aacfd23, State: Running, Role: LEADER
I20260925 16:37:26.769927 22900 consensus_queue.cc:237] T 00000000000000000000000000000000 P e305eb7d1f1c4c7da776df775aacfd23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e305eb7d1f1c4c7da776df775aacfd23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32819 } } peers { permanent_uuid: "aa46560c36b64aa2afe1776d54d06cb3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46101 } } peers { permanent_uuid: "6a9095c1d0f943648d6e9d91a1fc34fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46117 } }
W20260925 16:37:26.770732 22692 tablet.cc:2369] T 00000000000000000000000000000000 P e305eb7d1f1c4c7da776df775aacfd23: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 16:37:26.771787 22911 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aa46560c36b64aa2afe1776d54d06cb3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260925 16:37:26.771744 22900 sys_catalog.cc:455] T 00000000000000000000000000000000 P e305eb7d1f1c4c7da776df775aacfd23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e305eb7d1f1c4c7da776df775aacfd23. Latest consensus state: current_term: 1 leader_uuid: "e305eb7d1f1c4c7da776df775aacfd23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e305eb7d1f1c4c7da776df775aacfd23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32819 } } peers { permanent_uuid: "aa46560c36b64aa2afe1776d54d06cb3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46101 } } peers { permanent_uuid: "6a9095c1d0f943648d6e9d91a1fc34fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46117 } } }
I20260925 16:37:26.771862 22900 sys_catalog.cc:458] T 00000000000000000000000000000000 P e305eb7d1f1c4c7da776df775aacfd23 [sys.catalog]: This master's current role is: LEADER
W20260925 16:37:26.772035 22911 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 16:37:26.774796 22899 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa46560c36b64aa2afe1776d54d06cb3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e305eb7d1f1c4c7da776df775aacfd23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32819 } } peers { permanent_uuid: "aa46560c36b64aa2afe1776d54d06cb3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46101 } } peers { permanent_uuid: "6a9095c1d0f943648d6e9d91a1fc34fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46117 } } }
I20260925 16:37:26.774986 22899 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa46560c36b64aa2afe1776d54d06cb3 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:37:26.772936 22671 master_runner.cc:186] Error getting master registration for 127.0.98.60:46117: OK, instance_id { permanent_uuid: "6a9095c1d0f943648d6e9d91a1fc34fe" instance_seqno: 1790354246702553 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 16:37:26.776885 22671 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 16:37:26.777105 22890 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a9095c1d0f943648d6e9d91a1fc34fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e305eb7d1f1c4c7da776df775aacfd23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32819 } } peers { permanent_uuid: "aa46560c36b64aa2afe1776d54d06cb3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46101 } } peers { permanent_uuid: "6a9095c1d0f943648d6e9d91a1fc34fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46117 } }
I20260925 16:37:26.777392 22890 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a9095c1d0f943648d6e9d91a1fc34fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a9095c1d0f943648d6e9d91a1fc34fe, State: Initialized, Role: FOLLOWER
I20260925 16:37:26.777773 22890 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a9095c1d0f943648d6e9d91a1fc34fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e305eb7d1f1c4c7da776df775aacfd23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32819 } } peers { permanent_uuid: "aa46560c36b64aa2afe1776d54d06cb3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46101 } } peers { permanent_uuid: "6a9095c1d0f943648d6e9d91a1fc34fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46117 } }
I20260925 16:37:26.779191 22890 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a9095c1d0f943648d6e9d91a1fc34fe [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:37:26.779595 22864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e305eb7d1f1c4c7da776df775aacfd23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6a9095c1d0f943648d6e9d91a1fc34fe"
I20260925 16:37:26.779830 22864 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a9095c1d0f943648d6e9d91a1fc34fe [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:37:26.781277 22864 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6a9095c1d0f943648d6e9d91a1fc34fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e305eb7d1f1c4c7da776df775aacfd23 in term 1.
I20260925 16:37:26.784118 22896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:26.766797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56148 (local address 127.0.98.60:46117)
0925 16:37:26.766919 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:26.766925 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:26.767159 (+ 234us) server_negotiation.cc:407] Connection header received
0925 16:37:26.771416 (+ 4257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:26.771420 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:26.771470 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:26.771531 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:26.772588 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:26.773413 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:26.774560 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:26.775015 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:26.775105 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:26.777558 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:37:26.777562 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:37:26.777565 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:37:26.777590 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:37:26.779456 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:26.779958 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:26.779961 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:26.779963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:26.780046 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:26.780338 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:26.780341 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:26.780343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:26.780530 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:37:26.780679 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:26.781912 (+ 1233us) server_negotiation.cc:299] Negotiation successful
0925 16:37:26.782032 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 16:37:26.782568 22934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a9095c1d0f943648d6e9d91a1fc34fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e305eb7d1f1c4c7da776df775aacfd23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32819 } } peers { permanent_uuid: "aa46560c36b64aa2afe1776d54d06cb3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46101 } } peers { permanent_uuid: "6a9095c1d0f943648d6e9d91a1fc34fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46117 } } }
I20260925 16:37:26.788200 22934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a9095c1d0f943648d6e9d91a1fc34fe [sys.catalog]: This master's current role is: FOLLOWER
W20260925 16:37:26.787006 22947 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6a9095c1d0f943648d6e9d91a1fc34fe: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 16:37:26.789369 22947 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 16:37:26.790998 22742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:26.778916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37622 (local address 127.0.98.62:32819)
0925 16:37:26.779025 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:26.779029 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:26.781540 (+ 2511us) server_negotiation.cc:407] Connection header received
0925 16:37:26.781584 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:26.781586 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:26.781630 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:26.781748 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:26.782509 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:26.783195 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:26.784144 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:26.784513 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:26.784606 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:26.787093 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:37:26.787097 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:37:26.787100 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:37:26.787130 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:37:26.788858 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:26.789396 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:26.789400 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:26.789402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:26.789491 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:26.789805 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:26.789808 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:26.789810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:26.790043 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:37:26.790181 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:26.790748 (+ 567us) server_negotiation.cc:299] Negotiation successful
0925 16:37:26.790859 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 16:37:26.792910 22820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:26.767030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56626 (local address 127.0.98.61:46101)
0925 16:37:26.767153 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:26.767157 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:26.772789 (+ 5632us) server_negotiation.cc:407] Connection header received
0925 16:37:26.772953 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:26.772957 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:26.772998 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:26.773047 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:26.775068 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:26.775763 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:26.778397 (+ 2634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:26.778800 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:26.778891 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:26.789531 (+ 10640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:37:26.789535 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:37:26.789541 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:37:26.789569 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:37:26.791286 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:26.791748 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:26.791751 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:26.791753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:26.791838 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:26.792150 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:26.792152 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:26.792153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:26.792329 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:37:26.792420 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:26.792671 (+ 251us) server_negotiation.cc:299] Negotiation successful
0925 16:37:26.792766 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260925 16:37:26.799052 22922 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e305eb7d1f1c4c7da776df775aacfd23: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 16:37:26.799130 22922 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 16:37:26.799185 22933 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:37:26.800110 22933 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:37:26.805873 22864 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6a9095c1d0f943648d6e9d91a1fc34fe [term 1 FOLLOWER]: Refusing update from remote peer e305eb7d1f1c4c7da776df775aacfd23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:37:26.806607 22930 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e305eb7d1f1c4c7da776df775aacfd23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a9095c1d0f943648d6e9d91a1fc34fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:37:26.810779 22934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a9095c1d0f943648d6e9d91a1fc34fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader e305eb7d1f1c4c7da776df775aacfd23. Latest consensus state: current_term: 1 leader_uuid: "e305eb7d1f1c4c7da776df775aacfd23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e305eb7d1f1c4c7da776df775aacfd23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32819 } } peers { permanent_uuid: "aa46560c36b64aa2afe1776d54d06cb3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46101 } } peers { permanent_uuid: "6a9095c1d0f943648d6e9d91a1fc34fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46117 } } }
I20260925 16:37:26.810910 22934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a9095c1d0f943648d6e9d91a1fc34fe [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:37:26.811986 22788 raft_consensus.cc:1261] T 00000000000000000000000000000000 P aa46560c36b64aa2afe1776d54d06cb3 [term 1 FOLLOWER]: Refusing update from remote peer e305eb7d1f1c4c7da776df775aacfd23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:37:26.812580 22930 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e305eb7d1f1c4c7da776df775aacfd23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa46560c36b64aa2afe1776d54d06cb3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:37:26.816119 22899 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa46560c36b64aa2afe1776d54d06cb3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e305eb7d1f1c4c7da776df775aacfd23. Latest consensus state: current_term: 1 leader_uuid: "e305eb7d1f1c4c7da776df775aacfd23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e305eb7d1f1c4c7da776df775aacfd23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32819 } } peers { permanent_uuid: "aa46560c36b64aa2afe1776d54d06cb3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46101 } } peers { permanent_uuid: "6a9095c1d0f943648d6e9d91a1fc34fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46117 } } }
I20260925 16:37:26.816236 22899 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa46560c36b64aa2afe1776d54d06cb3 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:37:26.819950 22930 sys_catalog.cc:455] T 00000000000000000000000000000000 P e305eb7d1f1c4c7da776df775aacfd23 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e305eb7d1f1c4c7da776df775aacfd23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e305eb7d1f1c4c7da776df775aacfd23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32819 } } peers { permanent_uuid: "aa46560c36b64aa2afe1776d54d06cb3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46101 } } peers { permanent_uuid: "6a9095c1d0f943648d6e9d91a1fc34fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46117 } } }
I20260925 16:37:26.820062 22930 sys_catalog.cc:458] T 00000000000000000000000000000000 P e305eb7d1f1c4c7da776df775aacfd23 [sys.catalog]: This master's current role is: LEADER
I20260925 16:37:26.821043 22899 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa46560c36b64aa2afe1776d54d06cb3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e305eb7d1f1c4c7da776df775aacfd23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e305eb7d1f1c4c7da776df775aacfd23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32819 } } peers { permanent_uuid: "aa46560c36b64aa2afe1776d54d06cb3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46101 } } peers { permanent_uuid: "6a9095c1d0f943648d6e9d91a1fc34fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46117 } } }
I20260925 16:37:26.821166 22899 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa46560c36b64aa2afe1776d54d06cb3 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:37:26.821555 22949 mvcc.cc:205] Tried to move back new op lower bound from 7333290994908868608 to 7333290994773123072. Current Snapshot: MvccSnapshot[applied={T|T < 7333290994908868608}]
I20260925 16:37:26.824198 22900 sys_catalog.cc:455] T 00000000000000000000000000000000 P e305eb7d1f1c4c7da776df775aacfd23 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e305eb7d1f1c4c7da776df775aacfd23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e305eb7d1f1c4c7da776df775aacfd23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32819 } } peers { permanent_uuid: "aa46560c36b64aa2afe1776d54d06cb3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46101 } } peers { permanent_uuid: "6a9095c1d0f943648d6e9d91a1fc34fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46117 } } }
I20260925 16:37:26.824308 22900 sys_catalog.cc:458] T 00000000000000000000000000000000 P e305eb7d1f1c4c7da776df775aacfd23 [sys.catalog]: This master's current role is: LEADER
I20260925 16:37:26.825482 22934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a9095c1d0f943648d6e9d91a1fc34fe [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e305eb7d1f1c4c7da776df775aacfd23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e305eb7d1f1c4c7da776df775aacfd23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32819 } } peers { permanent_uuid: "aa46560c36b64aa2afe1776d54d06cb3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46101 } } peers { permanent_uuid: "6a9095c1d0f943648d6e9d91a1fc34fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46117 } } }
I20260925 16:37:26.825595 22934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a9095c1d0f943648d6e9d91a1fc34fe [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:37:26.826331 22950 mvcc.cc:205] Tried to move back new op lower bound from 7333290994908868608 to 7333290994773123072. Current Snapshot: MvccSnapshot[applied={T|T < 7333290994908868608}]
I20260925 16:37:26.828325 22933 catalog_manager.cc:1347] Generated new cluster ID: 264128a642a7455aa9711b9031b50b2e
I20260925 16:37:26.828387 22933 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:37:26.829296 22952 mvcc.cc:205] Tried to move back new op lower bound from 7333290994908868608 to 7333290994773123072. Current Snapshot: MvccSnapshot[applied={T|T < 7333290994908868608}]
I20260925 16:37:26.853214 22933 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:37:26.854732 22933 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:37:26.867304 22933 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e305eb7d1f1c4c7da776df775aacfd23: Generated new TSK 0
I20260925 16:37:26.868300 22933 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 16:37:26.902204 22896 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:37:26.898969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56154 (local address 127.0.98.60:46117)
0925 16:37:26.899092 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:26.899098 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:26.901021 (+ 1923us) server_negotiation.cc:407] Connection header received
0925 16:37:26.901247 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:26.901251 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:26.901303 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:26.901364 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:26.902065 (+ 701us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56154: BlockingRecv error: recv got EOF from 127.0.0.1:56154 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260925 16:37:26.903717 22820 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:37:26.899436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56642 (local address 127.0.98.61:46101)
0925 16:37:26.899546 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:26.899550 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:26.902889 (+ 3339us) server_negotiation.cc:407] Connection header received
0925 16:37:26.903052 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:26.903055 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:26.903102 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:26.903151 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:26.903603 (+ 452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56642: BlockingRecv error: recv got EOF from 127.0.0.1:56642 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260925 16:37:26.905172 22742 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:37:26.899860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37626 (local address 127.0.98.62:32819)
0925 16:37:26.899973 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:26.899977 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:26.904328 (+ 4351us) server_negotiation.cc:407] Connection header received
0925 16:37:26.904489 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:26.904492 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:26.904541 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:26.904589 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:26.905059 (+ 470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37626: BlockingRecv error: recv got EOF from 127.0.0.1:37626 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260925 16:37:26.909670 22820 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:37:26.908225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56652 (local address 127.0.98.61:46101)
0925 16:37:26.908324 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:26.908328 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:26.908785 (+ 457us) server_negotiation.cc:407] Connection header received
0925 16:37:26.908992 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:26.908996 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:26.909069 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:26.909117 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:26.909575 (+ 458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56652: BlockingRecv error: recv got EOF from 127.0.0.1:56652 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260925 16:37:26.914714 22820 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:37:26.913239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56660 (local address 127.0.98.61:46101)
0925 16:37:26.913347 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:26.913351 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:26.913859 (+ 508us) server_negotiation.cc:407] Connection header received
0925 16:37:26.914029 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:26.914032 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:26.914077 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:26.914125 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:26.914624 (+ 499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56660: BlockingRecv error: recv got EOF from 127.0.0.1:56660 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260925 16:37:26.920269 22820 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:37:26.918884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56666 (local address 127.0.98.61:46101)
0925 16:37:26.918970 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:26.918974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:26.919467 (+ 493us) server_negotiation.cc:407] Connection header received
0925 16:37:26.919632 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:26.919635 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:26.919681 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:26.919730 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:26.920177 (+ 447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56666: BlockingRecv error: recv got EOF from 127.0.0.1:56666 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260925 16:37:26.928102 22820 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:37:26.926523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56668 (local address 127.0.98.61:46101)
0925 16:37:26.926718 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:26.926725 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:26.927253 (+ 528us) server_negotiation.cc:407] Connection header received
0925 16:37:26.927429 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:26.927433 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:26.927477 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:26.927524 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:26.928008 (+ 484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56668: BlockingRecv error: recv got EOF from 127.0.0.1:56668 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
W20260925 16:37:26.940371 22820 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:37:26.938545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56684 (local address 127.0.98.61:46101)
0925 16:37:26.938724 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:26.938729 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:26.939329 (+ 600us) server_negotiation.cc:407] Connection header received
0925 16:37:26.939565 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:26.939568 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:26.939619 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:26.939672 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:26.940261 (+ 589us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56684: BlockingRecv error: recv got EOF from 127.0.0.1:56684 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
W20260925 16:37:26.961277 22820 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:37:26.959395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56696 (local address 127.0.98.61:46101)
0925 16:37:26.959517 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:26.959521 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:26.960185 (+ 664us) server_negotiation.cc:407] Connection header received
0925 16:37:26.960412 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:26.960417 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:26.960475 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:26.960534 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:26.961148 (+ 614us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56696: BlockingRecv error: recv got EOF from 127.0.0.1:56696 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260925 16:37:26.997367 22820 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:37:26.996027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56710 (local address 127.0.98.61:46101)
0925 16:37:26.996165 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:26.996170 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:26.996449 (+ 279us) server_negotiation.cc:407] Connection header received
0925 16:37:26.996621 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:26.996624 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:26.996679 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:26.996774 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:26.997239 (+ 465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56710: BlockingRecv error: recv got EOF from 127.0.0.1:56710 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260925 16:37:27.064790 22820 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:37:27.063501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56714 (local address 127.0.98.61:46101)
0925 16:37:27.063671 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:27.063676 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:27.063835 (+ 159us) server_negotiation.cc:407] Connection header received
0925 16:37:27.064012 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:27.064016 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:27.064069 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:27.064161 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:27.064660 (+ 499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56714: BlockingRecv error: recv got EOF from 127.0.0.1:56714 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260925 16:37:27.196095 22820 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:37:27.194860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56724 (local address 127.0.98.61:46101)
0925 16:37:27.195068 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:27.195071 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:27.195088 (+ 17us) server_negotiation.cc:407] Connection header received
0925 16:37:27.195249 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:27.195252 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:27.195301 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:27.195366 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:27.195935 (+ 569us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56724: BlockingRecv error: recv got EOF from 127.0.0.1:56724 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
W20260925 16:37:27.455117 22820 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:37:27.454006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56734 (local address 127.0.98.61:46101)
0925 16:37:27.454158 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:27.454162 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:27.454177 (+ 15us) server_negotiation.cc:407] Connection header received
0925 16:37:27.454324 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:27.454327 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:27.454373 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:27.454446 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:27.454959 (+ 513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56734: BlockingRecv error: recv got EOF from 127.0.0.1:56734 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 16:37:27.773447 22911 catalog_manager.cc:1259] Loaded cluster ID: 264128a642a7455aa9711b9031b50b2e
I20260925 16:37:27.773531 22911 catalog_manager.cc:1552] T 00000000000000000000000000000000 P aa46560c36b64aa2afe1776d54d06cb3: loading cluster ID for follower catalog manager: success
I20260925 16:37:27.775315 22911 catalog_manager.cc:1574] T 00000000000000000000000000000000 P aa46560c36b64aa2afe1776d54d06cb3: acquiring CA information for follower catalog manager: success
I20260925 16:37:27.776481 22911 catalog_manager.cc:1602] T 00000000000000000000000000000000 P aa46560c36b64aa2afe1776d54d06cb3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 16:37:27.790804 22947 catalog_manager.cc:1259] Loaded cluster ID: 264128a642a7455aa9711b9031b50b2e
I20260925 16:37:27.790900 22947 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6a9095c1d0f943648d6e9d91a1fc34fe: loading cluster ID for follower catalog manager: success
I20260925 16:37:27.792939 22947 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6a9095c1d0f943648d6e9d91a1fc34fe: acquiring CA information for follower catalog manager: success
I20260925 16:37:27.793955 22947 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6a9095c1d0f943648d6e9d91a1fc34fe: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 16:37:27.972779 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:27.969051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56744 (local address 127.0.98.61:46101)
0925 16:37:27.969278 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:27.969282 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:27.969298 (+ 16us) server_negotiation.cc:407] Connection header received
0925 16:37:27.969441 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:27.969443 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:27.969566 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:27.969674 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:37:27.969979 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:27.970499 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:27.971444 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:27.971663 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:27.971749 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:27.972029 (+ 280us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:37:27.972470 (+ 441us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:37:27.972556 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:27.972611 (+ 55us) server_negotiation.cc:299] Negotiation successful
0925 16:37:27.972656 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":78,"threads_started":1}
I20260925 16:37:27.979777 23028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:27.976122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56168 (local address 127.0.98.60:46117)
0925 16:37:27.976332 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:27.976336 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:27.976365 (+ 29us) server_negotiation.cc:407] Connection header received
0925 16:37:27.976518 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:27.976520 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:27.976625 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:27.976718 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:37:27.976910 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:27.977413 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:27.978239 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:27.978453 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:27.978530 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:27.978858 (+ 328us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:37:27.979261 (+ 403us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:37:27.979353 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:27.979467 (+ 114us) server_negotiation.cc:299] Negotiation successful
0925 16:37:27.979512 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":77,"threads_started":1}
I20260925 16:37:27.987238 23034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:27.982708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37628 (local address 127.0.98.62:32819)
0925 16:37:27.982988 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:27.982992 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:27.983270 (+ 278us) server_negotiation.cc:407] Connection header received
0925 16:37:27.983478 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:27.983481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:27.983635 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:27.983742 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:37:27.983952 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:27.984735 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:27.985557 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:27.985808 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:27.985916 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:27.986228 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:37:27.986555 (+ 327us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:37:27.986683 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:27.987017 (+ 334us) server_negotiation.cc:299] Negotiation successful
0925 16:37:27.987074 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":69,"threads_started":1}
I20260925 16:37:27.987234 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:27.982724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56752 (local address 127.0.98.61:46101)
0925 16:37:27.983060 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:27.983064 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:27.983210 (+ 146us) server_negotiation.cc:407] Connection header received
0925 16:37:27.983425 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:27.983429 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:27.983476 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:27.983551 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:37:27.984072 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:27.984568 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:27.986037 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:27.986322 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:27.986441 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:27.986706 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:37:27.986788 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:37:27.986865 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:27.987024 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 16:37:27.987090 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252}
I20260925 16:37:27.987608 23028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:27.983140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56178 (local address 127.0.98.60:46117)
0925 16:37:27.983271 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:27.983275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:27.983288 (+ 13us) server_negotiation.cc:407] Connection header received
0925 16:37:27.983330 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:27.983332 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:27.983381 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:27.983448 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:37:27.983864 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:27.984651 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:27.986011 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:27.986313 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:27.986410 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:27.986932 (+ 522us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:37:27.987014 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:37:27.987075 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:27.987412 (+ 337us) server_negotiation.cc:299] Negotiation successful
0925 16:37:27.987466 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 16:37:27.995992 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 22671
I20260925 16:37:28.002272 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 22746
I20260925 16:37:28.008008 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 22822
2026-09-25T16:37:28Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3071 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353966814071-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:37:28 dist-test-slave-65pf krb5kdc[23039](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:37:28 dist-test-slave-65pf krb5kdc[23039](info): set up 2 sockets
Sep 25 16:37:28 dist-test-slave-65pf krb5kdc[23039](info): commencing operation
krb5kdc: starting...
W20260925 16:37:29.043056 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353966814071-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353966814071-392-0/krb5kdc/test-user.keytab.
Sep 25 16:37:29 dist-test-slave-65pf krb5kdc[23039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:37:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:37:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 16:37:29.167825 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353966814071-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353966814071-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353966814071-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38473
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38473,127.0.98.61:39801,127.0.98.60:36767
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:37:29.276929 23052 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 16:37:29.277285 23052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:37:29.277356 23052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:37:29.277431 23052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:37:29.281337 23052 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:37:29.281433 23052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:37:29.281474 23052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:37:29.281533 23052 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:37:29.281574 23052 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:37:29.286757 23052 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353966814071-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353966814071-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38473,127.0.98.61:39801,127.0.98.60:36767
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353966814071-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23052
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353966814071-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:37:29.287819 23052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:37:29.288843 23052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:37:29.297544 23060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:37:29.297618 23058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:37:29.297726 23052 server_base.cc:1034] running on GCE node
W20260925 16:37:29.297531 23057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:37:29.298426 23052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:37:29.299590 23052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:37:29.300809 23052 hybrid_clock.cc:648] HybridClock initialized: now 1790354249300792 us; error 26 us; skew 500 ppm
Sep 25 16:37:29 dist-test-slave-65pf krb5kdc[23039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354249, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:37:29.303656 23052 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 16:37:29.304527 23052 webserver.cc:460] Webserver started at http://127.0.98.62:40559/ using document root <none> and password file <none>
I20260925 16:37:29.305102 23052 fs_manager.cc:359] Metadata directory not provided
I20260925 16:37:29.305197 23052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:37:29.305444 23052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:37:29.307512 23052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353966814071-392-0/minicluster-data/master-0/data/instance:
uuid: "3f00731248dc47098ffd2dd815c24ad4"
format_stamp: "Formatted at 2026-09-25 16:37:29 on dist-test-slave-65pf"
server_key: "fb0e9db73bb28d33ba498da86e84bcf8"
server_key_iv: "92118778d936519730927b03e6ce760d"
server_key_version: "encryptionkey@0"
I20260925 16:37:29.308079 23052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353966814071-392-0/minicluster-data/master-0/wal/instance:
uuid: "3f00731248dc47098ffd2dd815c24ad4"
format_stamp: "Formatted at 2026-09-25 16:37:29 on dist-test-slave-65pf"
server_key: "fb0e9db73bb28d33ba498da86e84bcf8"
server_key_iv: "92118778d936519730927b03e6ce760d"
server_key_version: "encryptionkey@0"
I20260925 16:37:29.311791 23052 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260925 16:37:29.314236 23067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:37:29.315680 23052 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 16:37:29.315845 23052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353966814071-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353966814071-392-0/minicluster-data/master-0/wal
uuid: "3f00731248dc47098ffd2dd815c24ad4"
format_stamp: "Formatted at 2026-09-25 16:37:29 on dist-test-slave-65pf"
server_key: "fb0e9db73bb28d33ba498da86e84bcf8"
server_key_iv: "92118778d936519730927b03e6ce760d"
server_key_version: "encryptionkey@0"
I20260925 16:37:29.315958 23052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353966814071-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353966814071-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353966814071-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:37:29.332333 23052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:37:29.335564 23052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:37:29.335783 23052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:37:29.342407 23052 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38473
I20260925 16:37:29.342705 23119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38473 every 8 connection(s)
I20260925 16:37:29.343405 23052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353966814071-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:37:29.344302 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 23052
I20260925 16:37:29.344415 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353966814071-392-0/minicluster-data/master-0/wal/instance
I20260925 16:37:29.344722 392 external_mini_cluster.cc:1428] Setting key d124b79d1198a7199063a78244ae96d2
I20260925 16:37:29.346776 23120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:37:29.350463 23120 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38473 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 25 16:37:29 dist-test-slave-65pf krb5kdc[23039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354249, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:37:29.367070 23123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:29.352809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38473 (local address 127.0.0.1:57056)
0925 16:37:29.353266 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0925 16:37:29.353301 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 16:37:29.354160 (+ 859us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:37:29.355018 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:37:29.355026 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:37:29.355345 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:37:29.355589 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:37:29.355601 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:37:29.357074 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:37:29.357080 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:37:29.358374 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:37:29.358388 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:37:29.359118 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:37:29.359125 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:37:29.359289 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:29.359894 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:37:29.359921 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:37:29.361708 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:37:29.363823 (+ 2115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:37:29.363832 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:37:29.363841 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:37:29.364168 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:37:29.364555 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:37:29.364558 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:37:29.364561 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:37:29.364675 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:37:29.365311 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:37:29.365317 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:37:29.365462 (+ 145us) client_negotiation.cc:769] Sending connection context
0925 16:37:29.365635 (+ 173us) client_negotiation.cc:240] Negotiation successful
0925 16:37:29.365982 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":36,"thread_start_us":111,"threads_started":1}
I20260925 16:37:29.367070 23124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:29.352809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57056 (local address 127.0.98.62:38473)
0925 16:37:29.353292 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:29.353309 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:29.353415 (+ 106us) server_negotiation.cc:407] Connection header received
0925 16:37:29.354453 (+ 1038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:29.354457 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:29.354811 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:29.354976 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:29.355756 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:29.356942 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:29.358532 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:29.358958 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:29.359069 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:29.361864 (+ 2795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:37:29.361871 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:37:29.361887 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:37:29.361906 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:37:29.363697 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:29.364305 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:29.364311 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:29.364312 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:29.364415 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:29.364774 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:29.364780 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:29.364782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:29.365136 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:37:29.365313 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:29.365709 (+ 396us) server_negotiation.cc:299] Negotiation successful
0925 16:37:29.366026 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":90,"server-negotiator.queue_time_us":362,"spinlock_wait_cycles":56576,"thread_start_us":164,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353966814071-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353966814071-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260925 16:37:29.369455 23120 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39801 } has no permanent_uuid. Determining permanent_uuid...
I20260925 16:37:29.370087 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353966814071-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353966814071-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353966814071-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353966814071-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:39801
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353966814071-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38473,127.0.98.61:39801,127.0.98.60:36767
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 16:37:29.370208 23123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:29.369906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39801 (local address 127.0.0.1:46384)
0925 16:37:29.370050 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 16:37:29.370131 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 16:37:29.370371 23071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39801: connect: Connection refused (error 111)
I20260925 16:37:29.371266 23123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:29.370997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39801 (local address 127.0.0.1:46400)
0925 16:37:29.371121 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 16:37:29.371180 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 16:37:29.371433 23120 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39801: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39801: connect: Connection refused (error 111)
I20260925 16:37:29.420675 23120 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39801 } attempt: 1
I20260925 16:37:29.421530 23123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:29.421195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39801 (local address 127.0.0.1:46402)
0925 16:37:29.421373 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 16:37:29.421442 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260925 16:37:29.422235 23123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:29.422017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39801 (local address 127.0.0.1:46406)
0925 16:37:29.422119 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0925 16:37:29.422167 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260925 16:37:29.422358 23120 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39801: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39801: connect: Connection refused (error 111)
W20260925 16:37:29.476986 23127 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 16:37:29.477246 23127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:37:29.477269 23127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:37:29.477380 23127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:37:29.481345 23127 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:37:29.481405 23127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:37:29.481424 23127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:37:29.481441 23127 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:37:29.481456 23127 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:37:29.486646 23127 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353966814071-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353966814071-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38473,127.0.98.61:39801,127.0.98.60:36767
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353966814071-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:39801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353966814071-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353966814071-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:37:29.487653 23127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:37:29.488641 23127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 16:37:29.490526 23120 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39801 } attempt: 2
I20260925 16:37:29.491385 23123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:29.491050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39801 (local address 127.0.0.1:46416)
0925 16:37:29.491232 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 16:37:29.491303 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260925 16:37:29.492111 23123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:29.491885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39801 (local address 127.0.0.1:46424)
0925 16:37:29.492002 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 16:37:29.492052 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 16:37:29.492264 23120 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39801: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39801: connect: Connection refused (error 111)
W20260925 16:37:29.497229 23133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:37:29.497268 23132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:37:29.497268 23135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:37:29.497460 23127 server_base.cc:1034] running on GCE node
I20260925 16:37:29.497999 23127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:37:29.499217 23127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:37:29.500417 23127 hybrid_clock.cc:648] HybridClock initialized: now 1790354249500401 us; error 28 us; skew 500 ppm
Sep 25 16:37:29 dist-test-slave-65pf krb5kdc[23039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354249, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:37:29.503235 23127 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260925 16:37:29.504053 23127 webserver.cc:460] Webserver started at http://127.0.98.61:36333/ using document root <none> and password file <none>
I20260925 16:37:29.504593 23127 fs_manager.cc:359] Metadata directory not provided
I20260925 16:37:29.504683 23127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:37:29.504925 23127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:37:29.507014 23127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353966814071-392-0/minicluster-data/master-1/data/instance:
uuid: "b5aad144273940cb995821042f4d192e"
format_stamp: "Formatted at 2026-09-25 16:37:29 on dist-test-slave-65pf"
server_key: "7b850c18282ffbe6fe6f8af51e91c8a9"
server_key_iv: "7c8a5b2a2e05ed69cc668972d36db6af"
server_key_version: "encryptionkey@0"
I20260925 16:37:29.507560 23127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353966814071-392-0/minicluster-data/master-1/wal/instance:
uuid: "b5aad144273940cb995821042f4d192e"
format_stamp: "Formatted at 2026-09-25 16:37:29 on dist-test-slave-65pf"
server_key: "7b850c18282ffbe6fe6f8af51e91c8a9"
server_key_iv: "7c8a5b2a2e05ed69cc668972d36db6af"
server_key_version: "encryptionkey@0"
I20260925 16:37:29.511117 23127 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 16:37:29.513399 23142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:37:29.514575 23127 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:37:29.514772 23127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353966814071-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353966814071-392-0/minicluster-data/master-1/wal
uuid: "b5aad144273940cb995821042f4d192e"
format_stamp: "Formatted at 2026-09-25 16:37:29 on dist-test-slave-65pf"
server_key: "7b850c18282ffbe6fe6f8af51e91c8a9"
server_key_iv: "7c8a5b2a2e05ed69cc668972d36db6af"
server_key_version: "encryptionkey@0"
I20260925 16:37:29.514881 23127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353966814071-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353966814071-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353966814071-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:37:29.530941 23127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:37:29.534080 23127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:37:29.534235 23127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:37:29.540948 23127 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:39801
I20260925 16:37:29.541236 23194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:39801 every 8 connection(s)
I20260925 16:37:29.541904 23127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353966814071-392-0/minicluster-data/master-1/data/info.pb
I20260925 16:37:29.544553 23195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:37:29.546087 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 23127
I20260925 16:37:29.546226 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353966814071-392-0/minicluster-data/master-1/wal/instance
I20260925 16:37:29.546504 392 external_mini_cluster.cc:1428] Setting key 51af26320205d1ccd445a0df34bbe283
I20260925 16:37:29.548676 23195 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38473 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 25 16:37:29 dist-test-slave-65pf krb5kdc[23039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354249, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:37:29.563372 23124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:29.551028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57064 (local address 127.0.98.62:38473)
0925 16:37:29.551216 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:29.551222 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:29.552017 (+ 795us) server_negotiation.cc:407] Connection header received
0925 16:37:29.552920 (+ 903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:29.552924 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:29.552985 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:29.553098 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:29.554163 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:29.554968 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:29.556149 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:29.556386 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:29.556469 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:29.559610 (+ 3141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:37:29.559613 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:37:29.559615 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:37:29.559641 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:37:29.561226 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:29.561764 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:29.561769 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:29.561771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:29.561853 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:29.562186 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:29.562191 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:29.562193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:29.562373 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:37:29.562482 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:29.563054 (+ 572us) server_negotiation.cc:299] Negotiation successful
0925 16:37:29.563195 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 16:37:29.564164 23198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:29.551378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38473 (local address 127.0.0.1:57064)
0925 16:37:29.551846 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0925 16:37:29.551896 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 16:37:29.552710 (+ 814us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:37:29.553232 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:37:29.553238 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:37:29.553558 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:37:29.554011 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:37:29.554023 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:37:29.555099 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:37:29.555102 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:37:29.556039 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:37:29.556049 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:37:29.556563 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:37:29.556567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:37:29.556913 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:29.557650 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:37:29.557678 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:37:29.559397 (+ 1719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:37:29.561373 (+ 1976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:37:29.561379 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:37:29.561393 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:37:29.561645 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:37:29.561948 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:37:29.561953 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:37:29.561955 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:37:29.562088 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:37:29.562480 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:37:29.562487 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:37:29.562807 (+ 320us) client_negotiation.cc:769] Sending connection context
0925 16:37:29.562996 (+ 189us) client_negotiation.cc:240] Negotiation successful
0925 16:37:29.563339 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":143,"threads_started":1}
I20260925 16:37:29.565791 23195 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39801 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 16:37:29 dist-test-slave-65pf krb5kdc[23039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354249, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353966814071-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353966814071-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260925 16:37:29.573827 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
/tmp/dist-test-taskWmh9MY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353966814071-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353966814071-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353966814071-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353966814071-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:36767
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353966814071-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38473,127.0.98.61:39801,127.0.98.60:36767
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353966814071-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353966814071-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353966814071-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 16:37:29.574429 23198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:29.566351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39801 (local address 127.0.0.1:46426)
0925 16:37:29.566748 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0925 16:37:29.566763 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:37:29.566853 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:37:29.567231 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:37:29.567234 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:37:29.567408 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:37:29.567491 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:37:29.567496 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:37:29.568425 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:37:29.568431 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:37:29.568949 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:37:29.568956 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:37:29.569528 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:37:29.569532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:37:29.569653 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:29.570055 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:37:29.570065 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:37:29.571254 (+ 1189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:37:29.572926 (+ 1672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:37:29.572929 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:37:29.572931 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:37:29.573144 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:37:29.573413 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:37:29.573417 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:37:29.573419 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:37:29.573500 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:37:29.573935 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:37:29.573938 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:37:29.574016 (+ 78us) client_negotiation.cc:769] Sending connection context
0925 16:37:29.574130 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 16:37:29.574259 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347}
I20260925 16:37:29.574453 23200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:29.566433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46426 (local address 127.0.98.61:39801)
0925 16:37:29.566742 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:29.566778 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:29.566820 (+ 42us) server_negotiation.cc:407] Connection header received
0925 16:37:29.566959 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:29.566965 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:29.567117 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:29.567253 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:29.567596 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:29.568316 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:29.569061 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:29.569422 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:29.569533 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:29.571381 (+ 1848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:37:29.571390 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:37:29.571404 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:37:29.571428 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:37:29.572809 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:29.573243 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:29.573248 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:29.573250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:29.573322 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:29.573586 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:29.573591 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:29.573593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:29.573826 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:37:29.573960 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:29.574203 (+ 243us) server_negotiation.cc:299] Negotiation successful
0925 16:37:29.574320 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":126,"threads_started":1}
I20260925 16:37:29.576191 23195 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36767 } has no permanent_uuid. Determining permanent_uuid...
I20260925 16:37:29.576881 23198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:29.576623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36767 (local address 127.0.0.1:48068)
0925 16:37:29.576734 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 16:37:29.576826 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 16:37:29.577070 23145 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36767: connect: Connection refused (error 111)
I20260925 16:37:29.577875 23198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:29.577664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36767 (local address 127.0.0.1:48072)
0925 16:37:29.577774 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0925 16:37:29.577828 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 16:37:29.578023 23195 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36767: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36767: connect: Connection refused (error 111)
I20260925 16:37:29.583426 23120 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39801 } attempt: 3
Sep 25 16:37:29 dist-test-slave-65pf krb5kdc[23039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354249, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 16:37:29.591935 23123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:29.583897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39801 (local address 127.0.0.1:46442)
0925 16:37:29.584024 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 16:37:29.584037 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:37:29.584117 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:37:29.584362 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:37:29.584364 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:37:29.584530 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:37:29.584631 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:37:29.584636 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:37:29.585703 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:37:29.585706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:37:29.586246 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:37:29.586254 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:37:29.586858 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:37:29.586861 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:37:29.586959 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:29.587471 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:37:29.587484 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:37:29.588652 (+ 1168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:37:29.590530 (+ 1878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:37:29.590533 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:37:29.590534 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:37:29.590806 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:37:29.591081 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:37:29.591083 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:37:29.591084 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:37:29.591157 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:37:29.591546 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:37:29.591548 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:37:29.591629 (+ 81us) client_negotiation.cc:769] Sending connection context
0925 16:37:29.591707 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 16:37:29.591810 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 16:37:29.592088 23200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:29.583960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46442 (local address 127.0.98.61:39801)
0925 16:37:29.584069 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:29.584073 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:29.584089 (+ 16us) server_negotiation.cc:407] Connection header received
0925 16:37:29.584207 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:29.584210 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:29.584257 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:29.584341 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:29.584750 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:29.585600 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:29.586350 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:29.586759 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:29.586857 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:29.588786 (+ 1929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:37:29.588790 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:37:29.588792 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:37:29.588817 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:37:29.590422 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:29.590896 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:29.590900 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:29.590902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:29.590995 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:29.591236 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:29.591239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:29.591241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:29.591463 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:37:29.591553 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:29.591785 (+ 232us) server_negotiation.cc:299] Negotiation successful
0925 16:37:29.591898 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 16:37:29.593230 23120 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36767 } has no permanent_uuid. Determining permanent_uuid...
I20260925 16:37:29.594180 23123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:29.593944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36767 (local address 127.0.0.1:48078)
0925 16:37:29.594065 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 16:37:29.594115 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260925 16:37:29.594882 23123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:29.594654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36767 (local address 127.0.0.1:48084)
0925 16:37:29.594796 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 16:37:29.594836 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260925 16:37:29.595079 23120 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36767: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36767: connect: Connection refused (error 111)
I20260925 16:37:29.626271 23120 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36767 } attempt: 1
I20260925 16:37:29.627264 23195 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36767 } attempt: 1
I20260925 16:37:29.627386 23123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:29.626979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36767 (local address 127.0.0.1:48100)
0925 16:37:29.627153 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 16:37:29.627279 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260925 16:37:29.628052 23198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:29.627774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36767 (local address 127.0.0.1:48104)
0925 16:37:29.627920 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 16:37:29.627979 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260925 16:37:29.628331 23123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:29.627980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36767 (local address 127.0.0.1:48120)
0925 16:37:29.628204 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0925 16:37:29.628258 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
W20260925 16:37:29.628464 23120 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36767: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36767: connect: Connection refused (error 111)
I20260925 16:37:29.628782 23198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:29.628573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36767 (local address 127.0.0.1:48130)
0925 16:37:29.628666 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0925 16:37:29.628727 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260925 16:37:29.628933 23195 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36767: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36767: connect: Connection refused (error 111)
W20260925 16:37:29.684808 23201 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 16:37:29.685091 23201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:37:29.685114 23201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:37:29.685212 23201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:37:29.689208 23201 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:37:29.689270 23201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:37:29.689288 23201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:37:29.689304 23201 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:37:29.689361 23201 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:37:29.694487 23201 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353966814071-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353966814071-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38473,127.0.98.61:39801,127.0.98.60:36767
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353966814071-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:36767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353966814071-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353966814071-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:37:29.695534 23201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:37:29.696506 23201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 16:37:29.697145 23195 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36767 } attempt: 2
I20260925 16:37:29.697955 23198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:29.697631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36767 (local address 127.0.0.1:48142)
0925 16:37:29.697807 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 16:37:29.697873 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260925 16:37:29.698685 23198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:29.698438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36767 (local address 127.0.0.1:48156)
0925 16:37:29.698550 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0925 16:37:29.698623 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 16:37:29.698824 23195 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36767: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36767: connect: Connection refused (error 111)
I20260925 16:37:29.703641 23120 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36767 } attempt: 2
I20260925 16:37:29.704526 23123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:29.704165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36767 (local address 127.0.0.1:48162)
0925 16:37:29.704339 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 16:37:29.704400 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260925 16:37:29.705418 23123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:29.705074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36767 (local address 127.0.0.1:48178)
0925 16:37:29.705247 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 16:37:29.705296 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260925 16:37:29.705332 23208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:37:29.705298 23207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:37:29.705302 23210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:37:29.705608 23120 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36767: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36767: connect: Connection refused (error 111)
I20260925 16:37:29.705664 23201 server_base.cc:1034] running on GCE node
I20260925 16:37:29.706248 23201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:37:29.707388 23201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:37:29.708577 23201 hybrid_clock.cc:648] HybridClock initialized: now 1790354249708562 us; error 24 us; skew 500 ppm
Sep 25 16:37:29 dist-test-slave-65pf krb5kdc[23039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354249, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:37:29.711162 23201 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260925 16:37:29.712047 23201 webserver.cc:460] Webserver started at http://127.0.98.60:35693/ using document root <none> and password file <none>
I20260925 16:37:29.712596 23201 fs_manager.cc:359] Metadata directory not provided
I20260925 16:37:29.712685 23201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:37:29.712904 23201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:37:29.714937 23201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353966814071-392-0/minicluster-data/master-2/data/instance:
uuid: "f0a35bd9482148d592292244a59b81d9"
format_stamp: "Formatted at 2026-09-25 16:37:29 on dist-test-slave-65pf"
server_key: "c813302a45fdfd5f5f8b1513dab9326c"
server_key_iv: "76522828291eeaa98601f63dd4814176"
server_key_version: "encryptionkey@0"
I20260925 16:37:29.715459 23201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353966814071-392-0/minicluster-data/master-2/wal/instance:
uuid: "f0a35bd9482148d592292244a59b81d9"
format_stamp: "Formatted at 2026-09-25 16:37:29 on dist-test-slave-65pf"
server_key: "c813302a45fdfd5f5f8b1513dab9326c"
server_key_iv: "76522828291eeaa98601f63dd4814176"
server_key_version: "encryptionkey@0"
I20260925 16:37:29.719105 23201 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 16:37:29.721593 23217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:37:29.722888 23201 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 16:37:29.723033 23201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353966814071-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353966814071-392-0/minicluster-data/master-2/wal
uuid: "f0a35bd9482148d592292244a59b81d9"
format_stamp: "Formatted at 2026-09-25 16:37:29 on dist-test-slave-65pf"
server_key: "c813302a45fdfd5f5f8b1513dab9326c"
server_key_iv: "76522828291eeaa98601f63dd4814176"
server_key_version: "encryptionkey@0"
I20260925 16:37:29.723147 23201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353966814071-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353966814071-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353966814071-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:37:29.742719 23201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:37:29.745740 23201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:37:29.745955 23201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:37:29.753192 23201 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:36767
I20260925 16:37:29.753499 23269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:36767 every 8 connection(s)
I20260925 16:37:29.754194 23201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353966814071-392-0/minicluster-data/master-2/data/info.pb
I20260925 16:37:29.756954 23270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:37:29.760317 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu as pid 23201
I20260925 16:37:29.760448 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmh9MY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353966814071-392-0/minicluster-data/master-2/wal/instance
I20260925 16:37:29.760747 392 external_mini_cluster.cc:1428] Setting key e2391a006fd7d77575a13f39f0931846
I20260925 16:37:29.760622 23270 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38473 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 16:37:29 dist-test-slave-65pf krb5kdc[23039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354249, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 25 16:37:29 dist-test-slave-65pf krb5kdc[23039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 16:37:29.773965 23124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:29.762875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57070 (local address 127.0.98.62:38473)
0925 16:37:29.762999 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:29.763003 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:29.763272 (+ 269us) server_negotiation.cc:407] Connection header received
0925 16:37:29.763786 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:29.763791 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:29.763893 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:29.764021 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:29.764934 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:29.765727 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:29.766706 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:29.766931 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:29.767081 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:29.769805 (+ 2724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:37:29.769808 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:37:29.769810 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:37:29.769834 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:37:29.771620 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:29.772205 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:29.772209 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:29.772210 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:29.772307 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:29.772687 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:29.772690 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:29.772692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:29.772941 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:37:29.773041 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:29.773537 (+ 496us) server_negotiation.cc:299] Negotiation successful
0925 16:37:29.773659 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 16:37:29.774721 23273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:29.762092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48194 (local address 127.0.98.60:36767)
0925 16:37:29.762503 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:29.762517 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:29.762559 (+ 42us) server_negotiation.cc:407] Connection header received
0925 16:37:29.763442 (+ 883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:29.763461 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:29.763701 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:29.763898 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:29.764719 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:29.765987 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:29.766887 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:29.767294 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:29.767416 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:29.770271 (+ 2855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:37:29.770276 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:37:29.770306 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:37:29.770331 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:37:29.772219 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:29.772699 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:29.772704 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:29.772706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:29.772783 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:29.773060 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:29.773063 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:29.773065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:29.773304 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:37:29.773487 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:29.773687 (+ 200us) server_negotiation.cc:299] Negotiation successful
0925 16:37:29.773820 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":116,"threads_started":1}
I20260925 16:37:29.774740 23274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:29.762777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38473 (local address 127.0.0.1:57070)
0925 16:37:29.763139 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0925 16:37:29.763189 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 16:37:29.763519 (+ 330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:37:29.764227 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:37:29.764233 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:37:29.764564 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:37:29.764800 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:37:29.764811 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:37:29.765833 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:37:29.765835 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:37:29.766554 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:37:29.766562 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:37:29.767111 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:37:29.767115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:37:29.767282 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:29.767904 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:37:29.767929 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:37:29.769656 (+ 1727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:37:29.771756 (+ 2100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:37:29.771763 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:37:29.771770 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:37:29.772104 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:37:29.772403 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:37:29.772409 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:37:29.772413 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:37:29.772594 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:37:29.773048 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:37:29.773054 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:37:29.773286 (+ 232us) client_negotiation.cc:769] Sending connection context
0925 16:37:29.773459 (+ 173us) client_negotiation.cc:240] Negotiation successful
0925 16:37:29.773775 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":83,"threads_started":1}
I20260925 16:37:29.776248 23270 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39801 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 16:37:29 dist-test-slave-65pf krb5kdc[23039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354249, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 25 16:37:29 dist-test-slave-65pf krb5kdc[23039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:37:29.784464 23274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:29.776891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39801 (local address 127.0.0.1:46456)
0925 16:37:29.777027 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 16:37:29.777042 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:37:29.777117 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:37:29.777371 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:37:29.777374 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:37:29.777537 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:37:29.777617 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:37:29.777622 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:37:29.778323 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:37:29.778329 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:37:29.778888 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:37:29.778896 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:37:29.779317 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:37:29.779320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:37:29.779414 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:29.779870 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:37:29.779886 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:37:29.781213 (+ 1327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:37:29.782988 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:37:29.782993 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:37:29.782998 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:37:29.783200 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:37:29.783466 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:37:29.783468 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:37:29.783469 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:37:29.783526 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:37:29.783963 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:37:29.783968 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:37:29.784067 (+ 99us) client_negotiation.cc:769] Sending connection context
0925 16:37:29.784174 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 16:37:29.784285 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260925 16:37:29.784477 23200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:29.777002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46456 (local address 127.0.98.61:39801)
0925 16:37:29.777126 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:29.777133 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:29.777152 (+ 19us) server_negotiation.cc:407] Connection header received
0925 16:37:29.777198 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:29.777203 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:29.777268 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:29.777360 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:29.777733 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:29.778220 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:29.779000 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:29.779213 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:29.779318 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:29.781381 (+ 2063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:37:29.781386 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:37:29.781389 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:37:29.781417 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:37:29.782875 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:29.783294 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:29.783298 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:29.783300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:29.783381 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:29.783606 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:29.783613 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:29.783616 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:29.783868 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:37:29.783939 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:29.784241 (+ 302us) server_negotiation.cc:299] Negotiation successful
0925 16:37:29.784363 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 16:37:29.785485 23124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:29.777289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57080 (local address 127.0.98.62:38473)
0925 16:37:29.777395 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:29.777398 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:29.777407 (+ 9us) server_negotiation.cc:407] Connection header received
0925 16:37:29.777432 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:29.777434 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:29.777462 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:29.777548 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:29.778108 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:29.778543 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:29.779300 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:29.779509 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:29.779630 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:29.782004 (+ 2374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:37:29.782007 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:37:29.782009 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:37:29.782029 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:37:29.783731 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:29.784172 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:29.784175 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:29.784179 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:29.784266 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:29.784553 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:29.784559 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:29.784562 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:29.784774 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:37:29.784998 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:29.785133 (+ 135us) server_negotiation.cc:299] Negotiation successful
0925 16:37:29.785271 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 16:37:29.785578 23270 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36767 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 16:37:29 dist-test-slave-65pf krb5kdc[23039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354249, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 16:37:29.789994 23195 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36767 } attempt: 3
Sep 25 16:37:29 dist-test-slave-65pf krb5kdc[23039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 16:37:29.793481 23274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:29.786039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36767 (local address 127.0.0.1:48196)
0925 16:37:29.786161 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 16:37:29.786176 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:37:29.786242 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:37:29.786454 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:37:29.786457 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:37:29.786692 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:37:29.786786 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:37:29.786794 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:37:29.787446 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:37:29.787449 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:37:29.787919 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:37:29.787926 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:37:29.788362 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:37:29.788365 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:37:29.788478 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:29.788898 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:37:29.788911 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:37:29.790131 (+ 1220us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:37:29.792008 (+ 1877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:37:29.792013 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:37:29.792015 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:37:29.792198 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:37:29.792491 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:37:29.792494 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:37:29.792496 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:37:29.792597 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:37:29.792989 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:37:29.792992 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:37:29.793073 (+ 81us) client_negotiation.cc:769] Sending connection context
0925 16:37:29.793191 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 16:37:29.793311 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":30}
I20260925 16:37:29.794294 23273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:29.786128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48196 (local address 127.0.98.60:36767)
0925 16:37:29.786265 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:29.786271 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:29.786298 (+ 27us) server_negotiation.cc:407] Connection header received
0925 16:37:29.786327 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:29.786329 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:29.786361 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:29.786445 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:29.786877 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:29.787338 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:29.788026 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:29.788262 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:29.788365 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:29.790285 (+ 1920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:37:29.790287 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:37:29.790289 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:37:29.790306 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:37:29.791905 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:29.792305 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:29.792311 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:29.792314 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:29.792388 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:29.792687 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:29.792690 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:29.792692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:29.792909 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:37:29.792999 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:29.794018 (+ 1019us) server_negotiation.cc:299] Negotiation successful
0925 16:37:29.794131 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":82}
I20260925 16:37:29.795652 23200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:29.786743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46466 (local address 127.0.98.61:39801)
0925 16:37:29.786849 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:29.786853 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:29.786865 (+ 12us) server_negotiation.cc:407] Connection header received
0925 16:37:29.786936 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:29.786939 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:29.786975 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:29.787063 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:29.787602 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:29.788052 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:29.788823 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:29.789139 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:29.789250 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:29.791989 (+ 2739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:37:29.791993 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:37:29.791995 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:37:29.792022 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:37:29.793833 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:29.794293 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:29.794299 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:29.794303 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:29.794386 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:29.794753 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:29.794758 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:29.794762 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:29.795014 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:37:29.795237 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:29.795375 (+ 138us) server_negotiation.cc:299] Negotiation successful
0925 16:37:29.795489 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
Sep 25 16:37:29 dist-test-slave-65pf krb5kdc[23039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354249, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 16:37:29.798249 23270 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0a35bd9482148d592292244a59b81d9: Bootstrap starting.
I20260925 16:37:29.800490 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:37:29.801229 23270 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0a35bd9482148d592292244a59b81d9: Neither blocks nor log segments found. Creating new log.
I20260925 16:37:29.801409 23198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:29.790371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36767 (local address 127.0.0.1:48200)
0925 16:37:29.790515 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 16:37:29.790527 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 16:37:29.790634 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:37:29.791450 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:37:29.791453 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:37:29.791612 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:37:29.791678 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:37:29.791682 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:37:29.793132 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:37:29.793136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:37:29.793875 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:37:29.793888 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:37:29.794362 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:37:29.794367 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:37:29.794462 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:29.794961 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:37:29.794975 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:37:29.796461 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:37:29.799347 (+ 2886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:37:29.799350 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:37:29.799352 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:37:29.799602 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:37:29.800363 (+ 761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:37:29.800367 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:37:29.800369 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:37:29.800459 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:37:29.800862 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:37:29.800865 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:37:29.800963 (+ 98us) client_negotiation.cc:769] Sending connection context
0925 16:37:29.801069 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 16:37:29.801239 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"spinlock_wait_cycles":768}
I20260925 16:37:29.801420 23275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:29.790865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48200 (local address 127.0.98.60:36767)
0925 16:37:29.791198 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:29.791207 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:29.791230 (+ 23us) server_negotiation.cc:407] Connection header received
0925 16:37:29.791267 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:29.791270 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:29.791309 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:29.791433 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:29.791797 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:29.793007 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:29.794003 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:29.794265 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:29.794363 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:29.796588 (+ 2225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:37:29.796591 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:37:29.796593 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:37:29.796617 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:37:29.799237 (+ 2620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:29.800127 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:29.800131 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:29.800132 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:29.800241 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:29.800553 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:29.800558 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:29.800560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:29.800761 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:37:29.800871 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:29.801145 (+ 274us) server_negotiation.cc:299] Negotiation successful
0925 16:37:29.801255 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":361,"threads_started":1}
I20260925 16:37:29.802120 23270 log.cc:824] T 00000000000000000000000000000000 P f0a35bd9482148d592292244a59b81d9: Log is configured to *not* fsync() on all Append() calls
I20260925 16:37:29.801810 23094 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "3f00731248dc47098ffd2dd815c24ad4"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57080
I20260925 16:37:29.804415 23270 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0a35bd9482148d592292244a59b81d9: No bootstrap required, opened a new log
I20260925 16:37:29.804431 23195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5aad144273940cb995821042f4d192e: Bootstrap starting.
I20260925 16:37:29.804791 23120 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36767 } attempt: 3
I20260925 16:37:29.806798 23195 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5aad144273940cb995821042f4d192e: Neither blocks nor log segments found. Creating new log.
I20260925 16:37:29.807479 23195 log.cc:824] T 00000000000000000000000000000000 P b5aad144273940cb995821042f4d192e: Log is configured to *not* fsync() on all Append() calls
I20260925 16:37:29.808465 23270 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0a35bd9482148d592292244a59b81d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f00731248dc47098ffd2dd815c24ad4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38473 } } peers { permanent_uuid: "b5aad144273940cb995821042f4d192e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39801 } } peers { permanent_uuid: "f0a35bd9482148d592292244a59b81d9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36767 } }
I20260925 16:37:29.808725 23270 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0a35bd9482148d592292244a59b81d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0a35bd9482148d592292244a59b81d9, State: Initialized, Role: FOLLOWER
I20260925 16:37:29.809058 23270 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0a35bd9482148d592292244a59b81d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f00731248dc47098ffd2dd815c24ad4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38473 } } peers { permanent_uuid: "b5aad144273940cb995821042f4d192e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39801 } } peers { permanent_uuid: "f0a35bd9482148d592292244a59b81d9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36767 } }
I20260925 16:37:29.809381 23195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5aad144273940cb995821042f4d192e: No bootstrap required, opened a new log
I20260925 16:37:29.809513 23278 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0a35bd9482148d592292244a59b81d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f00731248dc47098ffd2dd815c24ad4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38473 } } peers { permanent_uuid: "b5aad144273940cb995821042f4d192e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39801 } } peers { permanent_uuid: "f0a35bd9482148d592292244a59b81d9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36767 } } }
I20260925 16:37:29.809631 23278 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0a35bd9482148d592292244a59b81d9 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:37:29.810250 23270 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0a35bd9482148d592292244a59b81d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:37:29.813280 23195 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5aad144273940cb995821042f4d192e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f00731248dc47098ffd2dd815c24ad4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38473 } } peers { permanent_uuid: "b5aad144273940cb995821042f4d192e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39801 } } peers { permanent_uuid: "f0a35bd9482148d592292244a59b81d9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36767 } }
I20260925 16:37:29.813474 23195 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5aad144273940cb995821042f4d192e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5aad144273940cb995821042f4d192e, State: Initialized, Role: FOLLOWER
I20260925 16:37:29.813766 23195 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5aad144273940cb995821042f4d192e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f00731248dc47098ffd2dd815c24ad4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38473 } } peers { permanent_uuid: "b5aad144273940cb995821042f4d192e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39801 } } peers { permanent_uuid: "f0a35bd9482148d592292244a59b81d9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36767 } }
I20260925 16:37:29.814358 23279 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5aad144273940cb995821042f4d192e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f00731248dc47098ffd2dd815c24ad4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38473 } } peers { permanent_uuid: "b5aad144273940cb995821042f4d192e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39801 } } peers { permanent_uuid: "f0a35bd9482148d592292244a59b81d9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36767 } } }
I20260925 16:37:29.814652 23279 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5aad144273940cb995821042f4d192e [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:37:29.815375 23195 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5aad144273940cb995821042f4d192e [sys.catalog]: configured and running, proceeding with master startup.
Sep 25 16:37:29 dist-test-slave-65pf krb5kdc[23039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790354249, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
W20260925 16:37:29.819870 23290 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f0a35bd9482148d592292244a59b81d9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 16:37:29.819949 23290 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 16:37:29.817456 23201 master_runner.cc:186] Error getting master registration for 127.0.98.62:38473: OK, instance_id { permanent_uuid: "3f00731248dc47098ffd2dd815c24ad4" instance_seqno: 1790354249340663 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 16:37:29.821179 23201 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 16:37:29.822500 23123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:29.805162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36767 (local address 127.0.0.1:48212)
0925 16:37:29.805331 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 16:37:29.805346 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:37:29.805439 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:37:29.808198 (+ 2759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:37:29.808201 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:37:29.808388 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:37:29.808520 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:37:29.808528 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:37:29.814882 (+ 6354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:37:29.814888 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:37:29.815798 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:37:29.815811 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:37:29.816523 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:37:29.816530 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:37:29.816618 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:29.817144 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:37:29.817160 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:37:29.818685 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:37:29.820785 (+ 2100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:37:29.820803 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:37:29.820805 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:37:29.821042 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:37:29.821431 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:37:29.821434 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:37:29.821436 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:37:29.821535 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:37:29.821973 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:37:29.821978 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:37:29.822076 (+ 98us) client_negotiation.cc:769] Sending connection context
0925 16:37:29.822192 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 16:37:29.822333 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102}
I20260925 16:37:29.822542 23275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:29.805215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48212 (local address 127.0.98.60:36767)
0925 16:37:29.807953 (+ 2738us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:29.807957 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:29.807971 (+ 14us) server_negotiation.cc:407] Connection header received
0925 16:37:29.808010 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:29.808015 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:29.808074 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:29.808164 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:29.813688 (+ 5524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:29.814589 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:29.815947 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:29.816386 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:29.816860 (+ 474us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:29.818853 (+ 1993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:37:29.818857 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:37:29.818859 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:37:29.818883 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:37:29.820617 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:29.821163 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:29.821167 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:29.821169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:29.821301 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:29.821641 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:29.821644 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:29.821646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:29.821859 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:37:29.821955 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:29.822267 (+ 312us) server_negotiation.cc:299] Negotiation successful
0925 16:37:29.822390 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":2648}
W20260925 16:37:29.822916 23301 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b5aad144273940cb995821042f4d192e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 16:37:29.822978 23301 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 16:37:29.819352 23127 master_runner.cc:186] Error getting master registration for 127.0.98.62:38473: OK, instance_id { permanent_uuid: "3f00731248dc47098ffd2dd815c24ad4" instance_seqno: 1790354249340663 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 16:37:29.823304 23127 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 16:37:29.825160 23120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f00731248dc47098ffd2dd815c24ad4: Bootstrap starting.
I20260925 16:37:29.827647 23120 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f00731248dc47098ffd2dd815c24ad4: Neither blocks nor log segments found. Creating new log.
I20260925 16:37:29.828346 23120 log.cc:824] T 00000000000000000000000000000000 P 3f00731248dc47098ffd2dd815c24ad4: Log is configured to *not* fsync() on all Append() calls
I20260925 16:37:29.830469 23120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f00731248dc47098ffd2dd815c24ad4: No bootstrap required, opened a new log
I20260925 16:37:29.833468 23120 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f00731248dc47098ffd2dd815c24ad4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f00731248dc47098ffd2dd815c24ad4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38473 } } peers { permanent_uuid: "b5aad144273940cb995821042f4d192e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39801 } } peers { permanent_uuid: "f0a35bd9482148d592292244a59b81d9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36767 } }
I20260925 16:37:29.833685 23120 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f00731248dc47098ffd2dd815c24ad4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f00731248dc47098ffd2dd815c24ad4, State: Initialized, Role: FOLLOWER
I20260925 16:37:29.834019 23120 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f00731248dc47098ffd2dd815c24ad4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f00731248dc47098ffd2dd815c24ad4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38473 } } peers { permanent_uuid: "b5aad144273940cb995821042f4d192e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39801 } } peers { permanent_uuid: "f0a35bd9482148d592292244a59b81d9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36767 } }
I20260925 16:37:29.834529 23305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f00731248dc47098ffd2dd815c24ad4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f00731248dc47098ffd2dd815c24ad4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38473 } } peers { permanent_uuid: "b5aad144273940cb995821042f4d192e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39801 } } peers { permanent_uuid: "f0a35bd9482148d592292244a59b81d9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36767 } } }
I20260925 16:37:29.834780 23305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f00731248dc47098ffd2dd815c24ad4 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:37:29.835335 23120 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f00731248dc47098ffd2dd815c24ad4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:37:29.835383 23094 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f00731248dc47098ffd2dd815c24ad4 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 16:37:29.835471 23094 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f00731248dc47098ffd2dd815c24ad4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:37:29.836781 23094 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f00731248dc47098ffd2dd815c24ad4 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f00731248dc47098ffd2dd815c24ad4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38473 } } peers { permanent_uuid: "b5aad144273940cb995821042f4d192e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39801 } } peers { permanent_uuid: "f0a35bd9482148d592292244a59b81d9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36767 } }
W20260925 16:37:29.837224 23073 tablet.cc:2369] T 00000000000000000000000000000000 P 3f00731248dc47098ffd2dd815c24ad4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:37:29.837765 23094 leader_election.cc:288] T 00000000000000000000000000000000 P 3f00731248dc47098ffd2dd815c24ad4 [CANDIDATE]: Term 1 election: Requested vote from peers b5aad144273940cb995821042f4d192e (127.0.98.61:39801), f0a35bd9482148d592292244a59b81d9 (127.0.98.60:36767)
I20260925 16:37:29.838444 23169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3f00731248dc47098ffd2dd815c24ad4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b5aad144273940cb995821042f4d192e"
I20260925 16:37:29.838719 23169 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5aad144273940cb995821042f4d192e [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:37:29.839406 23244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3f00731248dc47098ffd2dd815c24ad4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f0a35bd9482148d592292244a59b81d9"
I20260925 16:37:29.839618 23244 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0a35bd9482148d592292244a59b81d9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:37:29.840050 23169 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b5aad144273940cb995821042f4d192e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f00731248dc47098ffd2dd815c24ad4 in term 1.
I20260925 16:37:29.840435 23071 leader_election.cc:302] T 00000000000000000000000000000000 P 3f00731248dc47098ffd2dd815c24ad4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f00731248dc47098ffd2dd815c24ad4, b5aad144273940cb995821042f4d192e; no voters:
I20260925 16:37:29.840869 23244 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f0a35bd9482148d592292244a59b81d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f00731248dc47098ffd2dd815c24ad4 in term 1.
I20260925 16:37:29.843873 23305 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f00731248dc47098ffd2dd815c24ad4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:37:29.844589 23305 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f00731248dc47098ffd2dd815c24ad4 [term 1 LEADER]: Becoming Leader. State: Replica: 3f00731248dc47098ffd2dd815c24ad4, State: Running, Role: LEADER
I20260925 16:37:29.844991 23305 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f00731248dc47098ffd2dd815c24ad4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f00731248dc47098ffd2dd815c24ad4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38473 } } peers { permanent_uuid: "b5aad144273940cb995821042f4d192e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39801 } } peers { permanent_uuid: "f0a35bd9482148d592292244a59b81d9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36767 } }
I20260925 16:37:29.846767 23305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f00731248dc47098ffd2dd815c24ad4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f00731248dc47098ffd2dd815c24ad4. Latest consensus state: current_term: 1 leader_uuid: "3f00731248dc47098ffd2dd815c24ad4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f00731248dc47098ffd2dd815c24ad4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38473 } } peers { permanent_uuid: "b5aad144273940cb995821042f4d192e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39801 } } peers { permanent_uuid: "f0a35bd9482148d592292244a59b81d9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36767 } } }
I20260925 16:37:29.846941 23305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f00731248dc47098ffd2dd815c24ad4 [sys.catalog]: This master's current role is: LEADER
I20260925 16:37:29.847389 23052 master_runner.cc:263] No leader master found from master 3f00731248dc47098ffd2dd815c24ad4
I20260925 16:37:29.847453 23052 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 16:37:29.852911 23124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:29.842768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57092 (local address 127.0.98.62:38473)
0925 16:37:29.843054 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:29.843058 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:29.843635 (+ 577us) server_negotiation.cc:407] Connection header received
0925 16:37:29.843826 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:29.843832 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:29.843881 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:29.843935 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:29.845004 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:29.845741 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:29.846877 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:29.847208 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:29.847284 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:29.849098 (+ 1814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:37:29.849102 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:37:29.849105 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:37:29.849140 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:37:29.850818 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:29.851289 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:29.851293 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:29.851295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:29.851403 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:29.852001 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:29.852006 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:29.852009 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:29.852211 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:37:29.852309 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:29.852653 (+ 344us) server_negotiation.cc:299] Negotiation successful
0925 16:37:29.852774 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216}
W20260925 16:37:29.855129 23316 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3f00731248dc47098ffd2dd815c24ad4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 16:37:29.855260 23316 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 16:37:29.855384 23326 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:37:29.856216 23326 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:37:29.862304 23200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:29.843291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46472 (local address 127.0.98.61:39801)
0925 16:37:29.843504 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:29.843508 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:29.847360 (+ 3852us) server_negotiation.cc:407] Connection header received
0925 16:37:29.847591 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:29.847595 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:29.847646 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:29.847717 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:29.848725 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:29.849459 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:29.850665 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:29.851001 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:29.851073 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:29.858050 (+ 6977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:37:29.858055 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:37:29.858057 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:37:29.858084 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:37:29.860097 (+ 2013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:29.860806 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:29.860812 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:29.860816 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:29.860903 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:29.861255 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:29.861260 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:29.861263 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:29.861508 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:37:29.861622 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:29.861982 (+ 360us) server_negotiation.cc:299] Negotiation successful
0925 16:37:29.862106 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146}
I20260925 16:37:29.863754 23275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:29.843394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48216 (local address 127.0.98.60:36767)
0925 16:37:29.843592 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:29.843598 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:29.854234 (+ 10636us) server_negotiation.cc:407] Connection header received
0925 16:37:29.854386 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:29.854388 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:29.854433 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:29.854479 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:29.855307 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:29.856022 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:29.858069 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:29.858430 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:29.858519 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:29.860226 (+ 1707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:37:29.860229 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:37:29.860233 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:37:29.860260 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:37:29.861939 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:29.862371 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:29.862375 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:29.862377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:29.862462 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:37:29.862822 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:37:29.862825 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:37:29.862827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:37:29.863039 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:37:29.863153 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:29.863472 (+ 319us) server_negotiation.cc:299] Negotiation successful
0925 16:37:29.863585 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
I20260925 16:37:29.870914 23244 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f0a35bd9482148d592292244a59b81d9 [term 1 FOLLOWER]: Refusing update from remote peer 3f00731248dc47098ffd2dd815c24ad4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:37:29.871733 23324 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3f00731248dc47098ffd2dd815c24ad4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a35bd9482148d592292244a59b81d9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:37:29.877316 23278 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0a35bd9482148d592292244a59b81d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f00731248dc47098ffd2dd815c24ad4. Latest consensus state: current_term: 1 leader_uuid: "3f00731248dc47098ffd2dd815c24ad4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f00731248dc47098ffd2dd815c24ad4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38473 } } peers { permanent_uuid: "b5aad144273940cb995821042f4d192e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39801 } } peers { permanent_uuid: "f0a35bd9482148d592292244a59b81d9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36767 } } }
I20260925 16:37:29.877451 23278 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0a35bd9482148d592292244a59b81d9 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:37:29.881955 23278 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0a35bd9482148d592292244a59b81d9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3f00731248dc47098ffd2dd815c24ad4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f00731248dc47098ffd2dd815c24ad4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38473 } } peers { permanent_uuid: "b5aad144273940cb995821042f4d192e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39801 } } peers { permanent_uuid: "f0a35bd9482148d592292244a59b81d9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36767 } } }
I20260925 16:37:29.882064 23278 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0a35bd9482148d592292244a59b81d9 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:37:29.884759 23169 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b5aad144273940cb995821042f4d192e [term 1 FOLLOWER]: Refusing update from remote peer 3f00731248dc47098ffd2dd815c24ad4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:37:29.885352 23305 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3f00731248dc47098ffd2dd815c24ad4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5aad144273940cb995821042f4d192e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:37:29.889398 23329 mvcc.cc:205] Tried to move back new op lower bound from 7333291007464267776 to 7333291007368605696. Current Snapshot: MvccSnapshot[applied={T|T < 7333291007464267776}]
I20260925 16:37:29.889910 23326 catalog_manager.cc:1347] Generated new cluster ID: df5c9567c44d41be9e476a361db64f8b
I20260925 16:37:29.889984 23326 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:37:29.890285 23279 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5aad144273940cb995821042f4d192e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f00731248dc47098ffd2dd815c24ad4. Latest consensus state: current_term: 1 leader_uuid: "3f00731248dc47098ffd2dd815c24ad4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f00731248dc47098ffd2dd815c24ad4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38473 } } peers { permanent_uuid: "b5aad144273940cb995821042f4d192e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39801 } } peers { permanent_uuid: "f0a35bd9482148d592292244a59b81d9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36767 } } }
I20260925 16:37:29.890416 23279 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5aad144273940cb995821042f4d192e [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:37:29.890508 23279 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5aad144273940cb995821042f4d192e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3f00731248dc47098ffd2dd815c24ad4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f00731248dc47098ffd2dd815c24ad4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38473 } } peers { permanent_uuid: "b5aad144273940cb995821042f4d192e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39801 } } peers { permanent_uuid: "f0a35bd9482148d592292244a59b81d9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36767 } } }
I20260925 16:37:29.890573 23279 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5aad144273940cb995821042f4d192e [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:37:29.890949 23328 mvcc.cc:205] Tried to move back new op lower bound from 7333291007464267776 to 7333291007368605696. Current Snapshot: MvccSnapshot[applied={T|T < 7333291007464267776 or (T in {7333291007464267776})}]
I20260925 16:37:29.891346 23305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f00731248dc47098ffd2dd815c24ad4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3f00731248dc47098ffd2dd815c24ad4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f00731248dc47098ffd2dd815c24ad4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38473 } } peers { permanent_uuid: "b5aad144273940cb995821042f4d192e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39801 } } peers { permanent_uuid: "f0a35bd9482148d592292244a59b81d9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36767 } } }
I20260925 16:37:29.891476 23305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f00731248dc47098ffd2dd815c24ad4 [sys.catalog]: This master's current role is: LEADER
I20260925 16:37:29.894531 23332 mvcc.cc:205] Tried to move back new op lower bound from 7333291007464267776 to 7333291007368605696. Current Snapshot: MvccSnapshot[applied={T|T < 7333291007464267776}]
I20260925 16:37:29.908496 23324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f00731248dc47098ffd2dd815c24ad4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3f00731248dc47098ffd2dd815c24ad4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f00731248dc47098ffd2dd815c24ad4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38473 } } peers { permanent_uuid: "b5aad144273940cb995821042f4d192e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39801 } } peers { permanent_uuid: "f0a35bd9482148d592292244a59b81d9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36767 } } }
I20260925 16:37:29.908651 23324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f00731248dc47098ffd2dd815c24ad4 [sys.catalog]: This master's current role is: LEADER
I20260925 16:37:29.912895 23326 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:37:29.914237 23326 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:37:29.927883 23326 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f00731248dc47098ffd2dd815c24ad4: Generated new TSK 0
I20260925 16:37:29.928812 23326 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 16:37:29.960476 23124 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:37:29.957013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57098 (local address 127.0.98.62:38473)
0925 16:37:29.957147 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:29.957151 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:29.959133 (+ 1982us) server_negotiation.cc:407] Connection header received
0925 16:37:29.959534 (+ 401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:29.959538 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:29.959592 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:29.959664 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:29.960328 (+ 664us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57098: BlockingRecv error: recv got EOF from 127.0.0.1:57098 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260925 16:37:29.961817 23200 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:37:29.957779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46474 (local address 127.0.98.61:39801)
0925 16:37:29.957898 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:29.957902 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:29.959450 (+ 1548us) server_negotiation.cc:407] Connection header received
0925 16:37:29.961039 (+ 1589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:29.961043 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:29.961092 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:29.961142 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:29.961659 (+ 517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46474: BlockingRecv error: recv got EOF from 127.0.0.1:46474 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260925 16:37:29.962944 23275 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:37:29.958443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48222 (local address 127.0.98.60:36767)
0925 16:37:29.958549 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:29.958554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:29.960862 (+ 2308us) server_negotiation.cc:407] Connection header received
0925 16:37:29.962171 (+ 1309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:29.962175 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:29.962227 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:29.962278 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:29.962828 (+ 550us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48222: BlockingRecv error: recv got EOF from 127.0.0.1:48222 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260925 16:37:29.967666 23200 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:37:29.966105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46484 (local address 127.0.98.61:39801)
0925 16:37:29.966206 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:29.966210 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:29.966740 (+ 530us) server_negotiation.cc:407] Connection header received
0925 16:37:29.966962 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:29.966965 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:29.967040 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:29.967090 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:29.967568 (+ 478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46484: BlockingRecv error: recv got EOF from 127.0.0.1:46484 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260925 16:37:29.972452 23200 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:37:29.971011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46486 (local address 127.0.98.61:39801)
0925 16:37:29.971122 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:29.971126 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:29.971621 (+ 495us) server_negotiation.cc:407] Connection header received
0925 16:37:29.971811 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:29.971813 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:29.971860 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:29.971909 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:29.972360 (+ 451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46486: BlockingRecv error: recv got EOF from 127.0.0.1:46486 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260925 16:37:29.977679 23200 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:37:29.976286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46492 (local address 127.0.98.61:39801)
0925 16:37:29.976377 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:29.976381 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:29.976870 (+ 489us) server_negotiation.cc:407] Connection header received
0925 16:37:29.977042 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:29.977045 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:29.977090 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:29.977138 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:29.977586 (+ 448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46492: BlockingRecv error: recv got EOF from 127.0.0.1:46492 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260925 16:37:29.985288 23200 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:37:29.983691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46506 (local address 127.0.98.61:39801)
0925 16:37:29.983828 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:29.983833 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:29.984379 (+ 546us) server_negotiation.cc:407] Connection header received
0925 16:37:29.984564 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:29.984567 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:29.984616 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:29.984665 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:29.985181 (+ 516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46506: BlockingRecv error: recv got EOF from 127.0.0.1:46506 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260925 16:37:29.997537 23200 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:37:29.995669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46508 (local address 127.0.98.61:39801)
0925 16:37:29.995803 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:29.995808 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:29.996420 (+ 612us) server_negotiation.cc:407] Connection header received
0925 16:37:29.996677 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:29.996683 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:29.996771 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:29.996839 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:29.997428 (+ 589us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46508: BlockingRecv error: recv got EOF from 127.0.0.1:46508 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260925 16:37:30.017961 23200 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:37:30.016060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46516 (local address 127.0.98.61:39801)
0925 16:37:30.016190 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:30.016195 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:30.016831 (+ 636us) server_negotiation.cc:407] Connection header received
0925 16:37:30.017059 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:30.017065 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:30.017124 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:30.017179 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:30.017842 (+ 663us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46516: BlockingRecv error: recv got EOF from 127.0.0.1:46516 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260925 16:37:30.053735 23200 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:37:30.052468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46518 (local address 127.0.98.61:39801)
0925 16:37:30.052607 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:30.052611 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:30.052858 (+ 247us) server_negotiation.cc:407] Connection header received
0925 16:37:30.052939 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:30.052943 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:30.053024 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:30.053144 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:30.053594 (+ 450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46518: BlockingRecv error: recv got EOF from 127.0.0.1:46518 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260925 16:37:30.120858 23200 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:37:30.119806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46530 (local address 127.0.98.61:39801)
0925 16:37:30.119941 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:30.119945 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:30.120134 (+ 189us) server_negotiation.cc:407] Connection header received
0925 16:37:30.120248 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:30.120251 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:30.120312 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:30.120386 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:30.120769 (+ 383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46530: BlockingRecv error: recv got EOF from 127.0.0.1:46530 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260925 16:37:30.252122 23200 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:37:30.250999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46544 (local address 127.0.98.61:39801)
0925 16:37:30.251171 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:30.251175 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:30.251265 (+ 90us) server_negotiation.cc:407] Connection header received
0925 16:37:30.251391 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:30.251411 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:30.251461 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:30.251536 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:30.251994 (+ 458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46544: BlockingRecv error: recv got EOF from 127.0.0.1:46544 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260925 16:37:30.511385 23200 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:37:30.510243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46558 (local address 127.0.98.61:39801)
0925 16:37:30.510398 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:30.510405 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:30.510439 (+ 34us) server_negotiation.cc:407] Connection header received
0925 16:37:30.510646 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:30.510650 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:30.510706 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:30.510824 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:37:30.511265 (+ 441us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46558: BlockingRecv error: recv got EOF from 127.0.0.1:46558 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 16:37:30.821257 23290 catalog_manager.cc:1259] Loaded cluster ID: df5c9567c44d41be9e476a361db64f8b
I20260925 16:37:30.821337 23290 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f0a35bd9482148d592292244a59b81d9: loading cluster ID for follower catalog manager: success
I20260925 16:37:30.823027 23290 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f0a35bd9482148d592292244a59b81d9: acquiring CA information for follower catalog manager: success
I20260925 16:37:30.824199 23301 catalog_manager.cc:1259] Loaded cluster ID: df5c9567c44d41be9e476a361db64f8b
I20260925 16:37:30.824265 23301 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b5aad144273940cb995821042f4d192e: loading cluster ID for follower catalog manager: success
I20260925 16:37:30.824509 23290 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f0a35bd9482148d592292244a59b81d9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 16:37:30.825785 23301 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b5aad144273940cb995821042f4d192e: acquiring CA information for follower catalog manager: success
I20260925 16:37:30.826890 23301 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b5aad144273940cb995821042f4d192e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 16:37:31.029100 23401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:31.025401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46566 (local address 127.0.98.61:39801)
0925 16:37:31.025678 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:31.025682 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:31.025698 (+ 16us) server_negotiation.cc:407] Connection header received
0925 16:37:31.025843 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:31.025845 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:31.025969 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:31.026064 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:37:31.026317 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:31.026948 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:31.027696 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:31.028005 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:31.028077 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:31.028331 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:37:31.028755 (+ 424us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:37:31.028841 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:31.028914 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 16:37:31.028982 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":88,"threads_started":1}
I20260925 16:37:31.035279 23407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:31.031890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48228 (local address 127.0.98.60:36767)
0925 16:37:31.032142 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:31.032147 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:31.032160 (+ 13us) server_negotiation.cc:407] Connection header received
0925 16:37:31.032279 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:31.032281 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:31.032389 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:31.032509 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:37:31.032745 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:31.033198 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:31.033948 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:31.034154 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:31.034256 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:31.034521 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:37:31.034926 (+ 405us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:37:31.035055 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:31.035090 (+ 35us) server_negotiation.cc:299] Negotiation successful
0925 16:37:31.035137 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":80,"threads_started":1}
I20260925 16:37:31.042063 23401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:31.038308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46582 (local address 127.0.98.61:39801)
0925 16:37:31.038414 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:31.038418 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:31.038604 (+ 186us) server_negotiation.cc:407] Connection header received
0925 16:37:31.038745 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:31.038749 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:31.038798 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:31.038858 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:37:31.039090 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:31.039598 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:31.040771 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:31.041090 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:31.041182 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:31.041567 (+ 385us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:37:31.041638 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:37:31.041706 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:31.041853 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 16:37:31.041905 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 16:37:31.042500 23413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:31.038043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57102 (local address 127.0.98.62:38473)
0925 16:37:31.038663 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:31.038668 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:31.038680 (+ 12us) server_negotiation.cc:407] Connection header received
0925 16:37:31.038759 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:31.038764 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:31.038900 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:31.039008 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:37:31.039256 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:31.039763 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:31.040832 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:31.041042 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:31.041150 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:31.041588 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:37:31.041884 (+ 296us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:37:31.041965 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:31.042307 (+ 342us) server_negotiation.cc:299] Negotiation successful
0925 16:37:31.042363 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":71,"threads_started":1}
I20260925 16:37:31.042706 23407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:37:31.038148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48234 (local address 127.0.98.60:36767)
0925 16:37:31.038257 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0925 16:37:31.038261 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:37:31.038648 (+ 387us) server_negotiation.cc:407] Connection header received
0925 16:37:31.038812 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:37:31.038817 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:37:31.038866 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:37:31.038924 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:37:31.039852 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:31.040302 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:31.041372 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:37:31.041703 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:37:31.041795 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:37:31.042100 (+ 305us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:37:31.042172 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:37:31.042259 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 16:37:31.042483 (+ 224us) server_negotiation.cc:299] Negotiation successful
0925 16:37:31.042528 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 16:37:31.045470 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 23052
I20260925 16:37:31.051961 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 23127
I20260925 16:37:31.057881 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmh9MY/build/debug/bin/kudu with pid 23201
2026-09-25T16:37:31Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3050 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6122 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (284235 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST